[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:44:56 dist-test-slave-2kcd krb5kdc[4973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:44:56 dist-test-slave-2kcd krb5kdc[4973](info): set up 2 sockets
Aug 08 04:44:56 dist-test-slave-2kcd krb5kdc[4973](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 04:44:57.868944 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.461s user 0.009s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:44:57 dist-test-slave-2kcd krb5kdc[4973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:44:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:44:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:44:57.992846 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:38453
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:38453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:44:58.114390 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:44:58.114643 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:44:58.114683 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:44:58.118615 4993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:44:58.118674 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:44:58.118712 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:44:58.118729 4993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:44:58.118744 4993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:44:58.123905 4993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:38453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:44:58.124868 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:44:58.125833 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:44:58.135416 5002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:44:58.135394 5005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:44:58.135357 5001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:44:58.135494 4993 server_base.cc:1034] running on GCE node
I20260808 04:44:58.136180 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:44:58.137465 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:44:58.138636 4993 hybrid_clock.cc:648] HybridClock initialized: now 1786164298138611 us; error 80 us; skew 500 ppm
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:44:58.141456 4993 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:44:58.142445 4993 webserver.cc:460] Webserver started at http://127.4.217.62:41949/ using document root <none> and password file <none>
I20260808 04:44:58.143087 4993 fs_manager.cc:359] Metadata directory not provided
I20260808 04:44:58.143184 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:44:58.143544 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:44:58.145776 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "e1f2207f8b284257a7aa34bac84b653f"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "4731659b273754a11719eff6ef03fe3b"
server_key_iv: "7fe5baa1dc772b1315d4965262b92aa5"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.146364 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "e1f2207f8b284257a7aa34bac84b653f"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "4731659b273754a11719eff6ef03fe3b"
server_key_iv: "7fe5baa1dc772b1315d4965262b92aa5"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.150496 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 04:44:58.152956 5014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.154258 4993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:44:58.154425 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "e1f2207f8b284257a7aa34bac84b653f"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "4731659b273754a11719eff6ef03fe3b"
server_key_iv: "7fe5baa1dc772b1315d4965262b92aa5"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.154548 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:44:58.175792 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:44:58.178846 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:44:58.179059 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:44:58.186991 4993 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:38453
I20260808 04:44:58.187031 5099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:38453 every 8 connection(s)
I20260808 04:44:58.188004 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:44:58.188948 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 4993
I20260808 04:44:58.189110 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:44:58.190413 4964 external_mini_cluster.cc:1428] Setting key 6d1b4fb10d1d7e8b3d33c5dcc529d411
I20260808 04:44:58.191216 5100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:44:58.198000 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f: Bootstrap starting.
I20260808 04:44:58.200716 5100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f: Neither blocks nor log segments found. Creating new log.
I20260808 04:44:58.201547 5100 log.cc:824] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:44:58.203578 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f: No bootstrap required, opened a new log
I20260808 04:44:58.207520 5100 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f2207f8b284257a7aa34bac84b653f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38453 } }
I20260808 04:44:58.207779 5100 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:44:58.207880 5100 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f2207f8b284257a7aa34bac84b653f, State: Initialized, Role: FOLLOWER
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:44:58.208380 5100 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f2207f8b284257a7aa34bac84b653f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38453 } }
I20260808 04:44:58.208580 5100 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:44:58.208674 5100 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:44:58.208806 5100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:44:58.209744 5100 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f2207f8b284257a7aa34bac84b653f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38453 } }
I20260808 04:44:58.210107 5100 leader_election.cc:302] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1f2207f8b284257a7aa34bac84b653f; no voters:
I20260808 04:44:58.210481 5100 leader_election.cc:288] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:44:58.210603 5111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:44:58.210793 5111 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [term 1 LEADER]: Becoming Leader. State: Replica: e1f2207f8b284257a7aa34bac84b653f, State: Running, Role: LEADER
I20260808 04:44:58.211087 5111 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f2207f8b284257a7aa34bac84b653f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38453 } }
I20260808 04:44:58.211805 5100 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:44:58.213461 5112 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1f2207f8b284257a7aa34bac84b653f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f2207f8b284257a7aa34bac84b653f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38453 } } }
I20260808 04:44:58.213609 5112 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [sys.catalog]: This master's current role is: LEADER
I20260808 04:44:58.214566 5113 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1f2207f8b284257a7aa34bac84b653f. Latest consensus state: current_term: 1 leader_uuid: "e1f2207f8b284257a7aa34bac84b653f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f2207f8b284257a7aa34bac84b653f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38453 } } }
I20260808 04:44:58.215378 5113 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f [sys.catalog]: This master's current role is: LEADER
I20260808 04:44:58.216606 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:44:58.193664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.4.217.62:38453)
0808 04:44:58.194061 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0808 04:44:58.194077 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:44:58.195411 (+ 1334us) server_negotiation.cc:407] Connection header received
0808 04:44:58.198815 (+ 3404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:44:58.198836 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:44:58.199228 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:44:58.199590 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:44:58.201765 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.202960 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.204258 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.204711 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.204922 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:44:58.209319 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:44:58.209326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:44:58.209347 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:44:58.209375 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:44:58.211471 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.212056 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.212062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.212067 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.212216 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.212668 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.212674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.212678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.213241 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:44:58.215349 (+ 2108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:44:58.215538 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 04:44:58.215774 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":122,"threads_started":1}
W20260808 04:44:58.217823 5134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:44:58.217911 5134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:44:58.218034 5122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:44:58.219025 5122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:44:58.225148 5122 catalog_manager.cc:1347] Generated new cluster ID: 9c0f20c57aae4ad8aa0c5a925f6ddc40
I20260808 04:44:58.225265 5122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:44:58.231278 5122 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:44:58.232434 5122 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:44:58.240329 5122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1f2207f8b284257a7aa34bac84b653f: Generated new TSK 0
I20260808 04:44:58.241106 5122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:44:58.281953 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38453
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:44:58.397627 5144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:44:58.397892 5144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:44:58.397970 5144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:44:58.401911 5144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:44:58.401996 5144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:44:58.402098 5144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:44:58.407214 5144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:44:58.408224 5144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:44:58.409269 5144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:44:58.419561 5151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:44:58.419662 5144 server_base.cc:1034] running on GCE node
W20260808 04:44:58.419569 5154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:44:58.419569 5152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:44:58.420182 5144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:44:58.420873 5144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:44:58.422036 5144 hybrid_clock.cc:648] HybridClock initialized: now 1786164298422002 us; error 47 us; skew 500 ppm
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:44:58.424736 5144 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:44:58.425621 5144 webserver.cc:460] Webserver started at http://127.4.217.1:36463/ using document root <none> and password file <none>
I20260808 04:44:58.426180 5144 fs_manager.cc:359] Metadata directory not provided
I20260808 04:44:58.426273 5144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:44:58.426515 5144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:44:58.428610 5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "3d7c8f4ab0a44e0d95267cd8f0fc71b8"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "65326f33f725bd422ae5b481ab0daf7f"
server_key_iv: "4dfacf3debf6692be081cb7e6afb7e06"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.429302 5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "3d7c8f4ab0a44e0d95267cd8f0fc71b8"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "65326f33f725bd422ae5b481ab0daf7f"
server_key_iv: "4dfacf3debf6692be081cb7e6afb7e06"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.433295 5144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:44:58.435644 5166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.436930 5144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:44:58.437026 5144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "3d7c8f4ab0a44e0d95267cd8f0fc71b8"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "65326f33f725bd422ae5b481ab0daf7f"
server_key_iv: "4dfacf3debf6692be081cb7e6afb7e06"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.437117 5144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:44:58.449739 5144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:44:58.452768 5144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:44:58.452977 5144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:44:58.453624 5144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:44:58.454397 5144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:44:58.454443 5144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.454480 5144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:44:58.454515 5144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.464582 5144 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:32937
I20260808 04:44:58.464663 5356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:32937 every 8 connection(s)
I20260808 04:44:58.465492 5144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:44:58.468585 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 5144
I20260808 04:44:58.468677 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:44:58.468978 4964 external_mini_cluster.cc:1428] Setting key 4f184519dd0f976800cf9eab81278555
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
I20260808 04:44:58.480965 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:44:58.467378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:58917 (local address 127.4.217.62:38453)
0808 04:44:58.467520 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:44:58.467524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:44:58.468330 (+ 806us) server_negotiation.cc:407] Connection header received
0808 04:44:58.470201 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:44:58.470203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:44:58.470247 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:44:58.470355 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:44:58.472629 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.473232 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.474356 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.474592 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.474709 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:44:58.477284 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:44:58.477287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:44:58.477289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:44:58.477312 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:44:58.478750 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.479322 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.479326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.479328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.479412 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.479765 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.479769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.479771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.479946 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:44:58.480063 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:44:58.480639 (+ 576us) server_negotiation.cc:299] Negotiation successful
0808 04:44:58.480763 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:44:58.481725 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:44:58.467740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38453 (local address 127.4.217.1:58917)
0808 04:44:58.468188 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 04:44:58.468229 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:44:58.469985 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:44:58.471613 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:44:58.471620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:44:58.471943 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:44:58.472474 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:44:58.472490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.473362 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.473366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:44:58.474223 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:44:58.474233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.474749 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.474754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:44:58.474921 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:44:58.475494 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:44:58.475511 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:44:58.477107 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:44:58.478877 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:44:58.478883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:44:58.478896 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:44:58.479217 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:44:58.479509 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:44:58.479513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:44:58.479516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:44:58.479665 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:44:58.480057 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:44:58.480064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:44:58.480380 (+ 316us) client_negotiation.cc:769] Sending connection context
0808 04:44:58.480590 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 04:44:58.480929 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260808 04:44:58.482988 5358 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38453
I20260808 04:44:58.483227 5358 heartbeater.cc:464] Registering TS with master...
I20260808 04:44:58.483718 5358 heartbeater.cc:511] Master 127.4.217.62:38453 requested a full tablet report, sending...
I20260808 04:44:58.485473 5044 ts_manager.cc:194] Registered new tserver with Master: 3d7c8f4ab0a44e0d95267cd8f0fc71b8 (127.4.217.1:32937)
I20260808 04:44:58.487288 5044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:58917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:44:58.497838 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38453
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:44:58.614379 5368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:44:58.614663 5368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:44:58.614748 5368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:44:58.618973 5368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:44:58.619041 5368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:44:58.619103 5368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:44:58.624429 5368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:44:58.625535 5368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:44:58.626618 5368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:44:58.636883 5384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:44:58.636979 5378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:44:58.636955 5368 server_base.cc:1034] running on GCE node
W20260808 04:44:58.636891 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:44:58.637526 5368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:44:58.638193 5368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:44:58.639384 5368 hybrid_clock.cc:648] HybridClock initialized: now 1786164298639372 us; error 28 us; skew 500 ppm
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:44:58.642303 5368 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:44:58.643157 5368 webserver.cc:460] Webserver started at http://127.4.217.2:45237/ using document root <none> and password file <none>
I20260808 04:44:58.643728 5368 fs_manager.cc:359] Metadata directory not provided
I20260808 04:44:58.643790 5368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:44:58.643996 5368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:44:58.645982 5368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "b9ee2d7a0f5c4a10905b8c1de7d1d2f6"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "085500fb24ee30b935de0bf5c771fcf5"
server_key_iv: "0aaeb63db6010a5919202602ea0a3f83"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.646574 5368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "b9ee2d7a0f5c4a10905b8c1de7d1d2f6"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "085500fb24ee30b935de0bf5c771fcf5"
server_key_iv: "0aaeb63db6010a5919202602ea0a3f83"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.650177 5368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:44:58.652509 5392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.653810 5368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:44:58.653908 5368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "b9ee2d7a0f5c4a10905b8c1de7d1d2f6"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "085500fb24ee30b935de0bf5c771fcf5"
server_key_iv: "0aaeb63db6010a5919202602ea0a3f83"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.653987 5368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:44:58.691982 5368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:44:58.695529 5368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:44:58.695727 5368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:44:58.696311 5368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:44:58.697163 5368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:44:58.697211 5368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.697254 5368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:44:58.697275 5368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.707638 5368 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:33473
I20260808 04:44:58.707664 5561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:33473 every 8 connection(s)
I20260808 04:44:58.708596 5368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:44:58.714634 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 5368
I20260808 04:44:58.714775 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:44:58.715065 4964 external_mini_cluster.cc:1428] Setting key 227f2ad10ec41a931ff421dfed5bd6df
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:44:58.724318 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:44:58.710463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:60083 (local address 127.4.217.62:38453)
0808 04:44:58.710621 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:44:58.710625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:44:58.711512 (+ 887us) server_negotiation.cc:407] Connection header received
0808 04:44:58.713028 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:44:58.713038 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:44:58.713128 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:44:58.713225 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:44:58.714491 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.715344 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.716714 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.717013 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.717099 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:44:58.720297 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:44:58.720303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:44:58.720309 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:44:58.720340 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:44:58.721903 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.722579 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.722582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.722583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.722661 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.723089 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.723091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.723092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.723267 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:44:58.723340 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:44:58.724081 (+ 741us) server_negotiation.cc:299] Negotiation successful
0808 04:44:58.724186 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:44:58.725636 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:44:58.710896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38453 (local address 127.4.217.2:60083)
0808 04:44:58.711360 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 04:44:58.711402 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:44:58.712749 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:44:58.713385 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:44:58.713392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:44:58.713754 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:44:58.714335 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:44:58.714350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.715503 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.715507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:44:58.716579 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:44:58.716591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.717172 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.717179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:44:58.717413 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:44:58.718126 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:44:58.718152 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:44:58.720116 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:44:58.722025 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:44:58.722033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:44:58.722052 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:44:58.722469 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:44:58.722763 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:44:58.722767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:44:58.722771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:44:58.722980 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:44:58.723378 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:44:58.723389 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:44:58.723787 (+ 398us) client_negotiation.cc:769] Sending connection context
0808 04:44:58.724044 (+ 257us) client_negotiation.cc:240] Negotiation successful
0808 04:44:58.724428 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:44:58.727035 5562 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38453
I20260808 04:44:58.727316 5562 heartbeater.cc:464] Registering TS with master...
I20260808 04:44:58.727867 5562 heartbeater.cc:511] Master 127.4.217.62:38453 requested a full tablet report, sending...
I20260808 04:44:58.729225 5044 ts_manager.cc:194] Registered new tserver with Master: b9ee2d7a0f5c4a10905b8c1de7d1d2f6 (127.4.217.2:33473)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
I20260808 04:44:58.730046 5044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:60083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:44:58.747215 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38453
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:44:58.864642 5569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:44:58.864902 5569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:44:58.864944 5569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:44:58.869095 5569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:44:58.869148 5569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:44:58.869247 5569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:44:58.874361 5569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:44:58.875365 5569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:44:58.876374 5569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:44:58.885942 5579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:44:58.885963 5578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:44:58.885948 5581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:44:58.886116 5569 server_base.cc:1034] running on GCE node
I20260808 04:44:58.886650 5569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:44:58.887283 5569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:44:58.888473 5569 hybrid_clock.cc:648] HybridClock initialized: now 1786164298888453 us; error 42 us; skew 500 ppm
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:44:58.891296 5569 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:44:58.892067 5569 webserver.cc:460] Webserver started at http://127.4.217.3:33451/ using document root <none> and password file <none>
I20260808 04:44:58.892561 5569 fs_manager.cc:359] Metadata directory not provided
I20260808 04:44:58.892620 5569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:44:58.892829 5569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:44:58.894778 5569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "11dbc386f1494f4583b25bf0d9accbeb"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "eda6e2d65c55474d42a5d11cc1d21d8f"
server_key_iv: "d8ebe29e1bdcf54f5366d8b1392880c2"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.895313 5569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "11dbc386f1494f4583b25bf0d9accbeb"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "eda6e2d65c55474d42a5d11cc1d21d8f"
server_key_iv: "d8ebe29e1bdcf54f5366d8b1392880c2"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.898911 5569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:44:58.901260 5592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.902420 5569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:44:58.902519 5569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "11dbc386f1494f4583b25bf0d9accbeb"
format_stamp: "Formatted at 2026-08-08 04:44:58 on dist-test-slave-2kcd"
server_key: "eda6e2d65c55474d42a5d11cc1d21d8f"
server_key_iv: "d8ebe29e1bdcf54f5366d8b1392880c2"
server_key_version: "encryptionkey@0"
I20260808 04:44:58.902601 5569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:44:58.914891 5569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:44:58.917565 5569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:44:58.917711 5569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:44:58.918306 5569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:44:58.919070 5569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:44:58.919117 5569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.919157 5569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:44:58.919178 5569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:44:58.930174 5569 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:42423
I20260808 04:44:58.930200 5781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:42423 every 8 connection(s)
I20260808 04:44:58.931010 5569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:44:58.934309 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 5569
I20260808 04:44:58.934453 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:44:58.934718 4964 external_mini_cluster.cc:1428] Setting key c78cc8fc767f6d67688ffb36ebf837a5
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:44:58.944235 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:44:58.932918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:60739 (local address 127.4.217.62:38453)
0808 04:44:58.933059 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:44:58.933063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:44:58.933851 (+ 788us) server_negotiation.cc:407] Connection header received
0808 04:44:58.935372 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:44:58.935375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:44:58.935422 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:44:58.935496 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:44:58.936821 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.937469 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.938565 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.938772 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.938851 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:44:58.941281 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:44:58.941284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:44:58.941286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:44:58.941309 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:44:58.942540 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.942985 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.942987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.942988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.943068 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.943355 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.943357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.943358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.943516 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:44:58.943592 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:44:58.944028 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 04:44:58.944120 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:44:58.944993 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:44:58.933276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38453 (local address 127.4.217.3:60739)
0808 04:44:58.933711 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0808 04:44:58.933752 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:44:58.935130 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:44:58.935710 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:44:58.935720 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:44:58.936097 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:44:58.936657 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:44:58.936670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.937616 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.937619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:44:58.938460 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:44:58.938469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.938900 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.938903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:44:58.939045 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:44:58.939624 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:44:58.939641 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:44:58.941133 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:44:58.942645 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:44:58.942649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:44:58.942660 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:44:58.942896 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:44:58.943148 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:44:58.943150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:44:58.943152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:44:58.943275 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:44:58.943611 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:44:58.943616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:44:58.943821 (+ 205us) client_negotiation.cc:769] Sending connection context
0808 04:44:58.943968 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 04:44:58.944191 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260808 04:44:58.946017 5782 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38453
I20260808 04:44:58.946244 5782 heartbeater.cc:464] Registering TS with master...
I20260808 04:44:58.946702 5782 heartbeater.cc:511] Master 127.4.217.62:38453 requested a full tablet report, sending...
I20260808 04:44:58.947731 5044 ts_manager.cc:194] Registered new tserver with Master: 11dbc386f1494f4583b25bf0d9accbeb (127.4.217.3:42423)
I20260808 04:44:58.948354 5044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:60739
I20260808 04:44:58.949218 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:44:58.977306 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:44:58.968239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.4.217.1:32937)
0808 04:44:58.968594 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0808 04:44:58.968608 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:44:58.968632 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:44:58.968681 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:44:58.968686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:44:58.968924 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:44:58.969045 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:44:58.969674 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.970424 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.971243 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.971594 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:58.971675 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:44:58.973937 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:44:58.973946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:44:58.973959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:44:58.973982 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:44:58.975646 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.976010 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.976015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.976017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.976096 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:58.976321 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:58.976323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:58.976324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:58.976560 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:44:58.976723 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:44:58.976936 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 04:44:58.977104 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":126,"threads_started":1}
W20260808 04:44:58.978302 5216 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35842
Aug 08 04:44:58 dist-test-slave-2kcd krb5kdc[4973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:44:59 dist-test-slave-2kcd krb5kdc[4973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:44:59.005856 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:44:58.997586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.4.217.1:32937)
0808 04:44:58.997746 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:44:58.997751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:44:58.997877 (+ 126us) server_negotiation.cc:407] Connection header received
0808 04:44:58.998062 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:44:58.998065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:44:58.998106 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:44:58.998194 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:44:58.998878 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:58.999390 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:59.000236 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:44:59.000472 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:44:59.000596 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:44:59.002908 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:44:59.002912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:44:59.002914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:44:59.002937 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:44:59.004167 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:59.004606 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:59.004609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:59.004610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:59.004687 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:44:59.004993 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:44:59.004996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:44:59.004998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:44:59.005163 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:44:59.005306 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:44:59.005552 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 04:44:59.005676 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:44:59.006966 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 5144
I20260808 04:44:59.041078 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 5368
I20260808 04:44:59.057933 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 5569
I20260808 04:44:59.072947 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 4993
2026-08-08T04:44:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2704 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:44:59 dist-test-slave-2kcd krb5kdc[5803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:44:59 dist-test-slave-2kcd krb5kdc[5803](info): set up 2 sockets
Aug 08 04:44:59 dist-test-slave-2kcd krb5kdc[5803](info): commencing operation
krb5kdc: starting...
W20260808 04:45:00.372144 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:00 dist-test-slave-2kcd krb5kdc[5803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:00.497162 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:38441
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:38441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:00.612277 5822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:00.612532 5822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:00.612572 5822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:00.616484 5822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:00.616546 5822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:00.616565 5822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:00.616581 5822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:00.616598 5822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:00.621709 5822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:38441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:38441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:00.622743 5822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:00.623716 5822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:00.632200 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:00.632238 5831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:00.632377 5835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:00.632450 5822 server_base.cc:1034] running on GCE node
I20260808 04:45:00.633774 5822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:00.635236 5822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:00.636449 5822 hybrid_clock.cc:648] HybridClock initialized: now 1786164300636435 us; error 45 us; skew 500 ppm
Aug 08 04:45:00 dist-test-slave-2kcd krb5kdc[5803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164300, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:00.640410 5822 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:00.642943 5822 webserver.cc:460] Webserver started at http://127.4.217.62:46275/ using document root <none> and password file <none>
I20260808 04:45:00.643695 5822 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:00.643782 5822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:00.644073 5822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:00.647049 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "80bd5863f27141fba0d2f35a03c667bb"
format_stamp: "Formatted at 2026-08-08 04:45:00 on dist-test-slave-2kcd"
server_key: "c88757acbcd5f561fc2ea9b76520cfbd"
server_key_iv: "8775a9e82d50f460397b88f3c63eed66"
server_key_version: "encryptionkey@0"
I20260808 04:45:00.647832 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "80bd5863f27141fba0d2f35a03c667bb"
format_stamp: "Formatted at 2026-08-08 04:45:00 on dist-test-slave-2kcd"
server_key: "c88757acbcd5f561fc2ea9b76520cfbd"
server_key_iv: "8775a9e82d50f460397b88f3c63eed66"
server_key_version: "encryptionkey@0"
I20260808 04:45:00.654047 5822 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260808 04:45:00.657030 5850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:00.658614 5822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:45:00.658751 5822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "80bd5863f27141fba0d2f35a03c667bb"
format_stamp: "Formatted at 2026-08-08 04:45:00 on dist-test-slave-2kcd"
server_key: "c88757acbcd5f561fc2ea9b76520cfbd"
server_key_iv: "8775a9e82d50f460397b88f3c63eed66"
server_key_version: "encryptionkey@0"
I20260808 04:45:00.658888 5822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:00.674430 5822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:00.678884 5822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:00.679101 5822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:00.687249 5822 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:38441
I20260808 04:45:00.687282 5935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:38441 every 8 connection(s)
I20260808 04:45:00.688359 5822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:00.691406 5937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:00.693611 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 5822
I20260808 04:45:00.693708 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:00.694003 4964 external_mini_cluster.cc:1428] Setting key e2ad7d8696ffdf4bd604839d4f0ae597
I20260808 04:45:00.698717 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb: Bootstrap starting.
I20260808 04:45:00.701542 5937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:00.702450 5937 log.cc:824] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:45:00 dist-test-slave-2kcd krb5kdc[5803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:00.704846 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb: No bootstrap required, opened a new log
I20260808 04:45:00.709949 5937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd5863f27141fba0d2f35a03c667bb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38441 } }
I20260808 04:45:00.710220 5937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:00.710284 5937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd5863f27141fba0d2f35a03c667bb, State: Initialized, Role: FOLLOWER
I20260808 04:45:00.710568 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:00.695475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.4.217.62:38441)
0808 04:45:00.695901 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:00.695920 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:00.695966 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:45:00.696760 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:00.696784 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:00.697254 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:00.697587 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:00.698913 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:00.700426 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:00.701438 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:00.702178 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:00.702360 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:00.705233 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:00.705239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:00.705256 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:00.705288 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:00.707327 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:00.707776 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:00.707783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:00.707789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:00.707937 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:00.708211 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:00.708214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:00.708216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:00.708759 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:00.708997 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:00.709260 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 04:45:00.709577 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":109,"threads_started":1}
I20260808 04:45:00.710776 5937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd5863f27141fba0d2f35a03c667bb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38441 } }
I20260808 04:45:00.710935 5937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:00.711002 5937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:00.711099 5937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:00.712172 5937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd5863f27141fba0d2f35a03c667bb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38441 } }
I20260808 04:45:00.712525 5937 leader_election.cc:302] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80bd5863f27141fba0d2f35a03c667bb; no voters:
I20260808 04:45:00.712935 5937 leader_election.cc:288] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:00.713073 5947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:00.713308 5947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [term 1 LEADER]: Becoming Leader. State: Replica: 80bd5863f27141fba0d2f35a03c667bb, State: Running, Role: LEADER
I20260808 04:45:00.713614 5947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd5863f27141fba0d2f35a03c667bb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38441 } }
I20260808 04:45:00.714224 5937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:00.715260 5948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80bd5863f27141fba0d2f35a03c667bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd5863f27141fba0d2f35a03c667bb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38441 } } }
I20260808 04:45:00.715385 5948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:00.715714 5950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80bd5863f27141fba0d2f35a03c667bb. Latest consensus state: current_term: 1 leader_uuid: "80bd5863f27141fba0d2f35a03c667bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd5863f27141fba0d2f35a03c667bb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38441 } } }
I20260808 04:45:00.715814 5950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:00.715907 5962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:00.718967 5962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:00.724591 5962 catalog_manager.cc:1347] Generated new cluster ID: 0d5dc0b0a1b94925afe5342c9a2265d4
I20260808 04:45:00.724653 5962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:00.752206 5962 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:00.753211 5962 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:00.763693 5962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80bd5863f27141fba0d2f35a03c667bb: Generated new TSK 0
I20260808 04:45:00.764478 5962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:45:00.797559 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38441
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:00.912367 5983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:00.912597 5983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:00.912665 5983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:00.917039 5983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:00.917091 5983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:00.917186 5983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:00.922310 5983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38441
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:00.923352 5983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:00.924335 5983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:45:00.934120 5983 server_base.cc:1034] running on GCE node
W20260808 04:45:00.933998 5989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:00.933997 5991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:00.934006 5993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:00.934727 5983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:00.935339 5983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:00.936527 5983 hybrid_clock.cc:648] HybridClock initialized: now 1786164300936514 us; error 25 us; skew 500 ppm
Aug 08 04:45:00 dist-test-slave-2kcd krb5kdc[5803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164300, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:00.939244 5983 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:45:00.940105 5983 webserver.cc:460] Webserver started at http://127.4.217.1:41943/ using document root <none> and password file <none>
I20260808 04:45:00.940670 5983 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:00.940764 5983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:00.941072 5983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:00.943035 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "19d7f0dab60444d98d0df5885b139e09"
format_stamp: "Formatted at 2026-08-08 04:45:00 on dist-test-slave-2kcd"
server_key: "e405f50a6dfcb7e966518257d58dbaf5"
server_key_iv: "4932584a350086df90db9305f438c54b"
server_key_version: "encryptionkey@0"
I20260808 04:45:00.943617 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "19d7f0dab60444d98d0df5885b139e09"
format_stamp: "Formatted at 2026-08-08 04:45:00 on dist-test-slave-2kcd"
server_key: "e405f50a6dfcb7e966518257d58dbaf5"
server_key_iv: "4932584a350086df90db9305f438c54b"
server_key_version: "encryptionkey@0"
I20260808 04:45:00.947255 5983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:45:00.949607 6004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:00.950749 5983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:00.950873 5983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "19d7f0dab60444d98d0df5885b139e09"
format_stamp: "Formatted at 2026-08-08 04:45:00 on dist-test-slave-2kcd"
server_key: "e405f50a6dfcb7e966518257d58dbaf5"
server_key_iv: "4932584a350086df90db9305f438c54b"
server_key_version: "encryptionkey@0"
I20260808 04:45:00.950977 5983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:00.969652 5983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:00.972823 5983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:00.972993 5983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:00.973598 5983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:00.974768 5983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:00.974818 5983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:00.974884 5983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:00.974916 5983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:00.984359 5983 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:40271
I20260808 04:45:00.984433 6180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:40271 every 8 connection(s)
I20260808 04:45:00.985280 5983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:00.994168 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 5983
I20260808 04:45:00.994289 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:00.994596 4964 external_mini_cluster.cc:1428] Setting key ce2fdf2047d69dc34c7ba87dffa790df
Aug 08 04:45:00 dist-test-slave-2kcd krb5kdc[5803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164300, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:00.999320 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:00.987039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:35401 (local address 127.4.217.62:38441)
0808 04:45:00.987185 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:00.987189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:00.987974 (+ 785us) server_negotiation.cc:407] Connection header received
0808 04:45:00.989389 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:00.989392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:00.989440 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:00.989516 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:00.990654 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:00.991172 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:00.992308 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:00.992546 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:00.992621 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:00.995443 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:00.995447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:00.995450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:00.995474 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:00.997152 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:00.997638 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:00.997645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:00.997650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:00.997751 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:00.998069 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:00.998075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:00.998079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:00.998339 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:00.998458 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:00.999093 (+ 635us) server_negotiation.cc:299] Negotiation successful
0808 04:45:00.999200 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:45:01.000471 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:00.987381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38441 (local address 127.4.217.1:35401)
0808 04:45:00.987827 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:00.987868 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:00.989193 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:00.989683 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:00.989690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:00.990048 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:00.990526 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:00.990539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:00.991279 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:00.991282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:00.992201 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:00.992211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:00.992676 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:00.992678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:00.992888 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:00.993539 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:00.993555 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:00.995238 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:00.997271 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:00.997276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:00.997287 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:00.997534 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:00.997846 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:00.997848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:00.997850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:00.997976 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:00.998482 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:00.998492 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:00.998818 (+ 326us) client_negotiation.cc:769] Sending connection context
0808 04:45:00.999055 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 04:45:00.999405 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
I20260808 04:45:01.001730 6181 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38441
I20260808 04:45:01.001993 6181 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:01.002509 6181 heartbeater.cc:511] Master 127.4.217.62:38441 requested a full tablet report, sending...
I20260808 04:45:01.004076 5866 ts_manager.cc:194] Registered new tserver with Master: 19d7f0dab60444d98d0df5885b139e09 (127.4.217.1:40271)
I20260808 04:45:01.005308 5866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:35401
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:45:01.023977 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38441
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:01.142417 6193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:01.142664 6193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:01.142707 6193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:01.146632 6193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:01.146689 6193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:01.146746 6193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:01.151949 6193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38441
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:01.152943 6193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:01.153952 6193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:01.163700 6202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:01.163780 6206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:01.163782 6204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:01.164038 6193 server_base.cc:1034] running on GCE node
I20260808 04:45:01.164549 6193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:01.165268 6193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:01.166469 6193 hybrid_clock.cc:648] HybridClock initialized: now 1786164301166456 us; error 34 us; skew 500 ppm
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[5803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164301, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:01.169332 6193 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:45:01.170209 6193 webserver.cc:460] Webserver started at http://127.4.217.2:36183/ using document root <none> and password file <none>
I20260808 04:45:01.170742 6193 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:01.170800 6193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:01.170964 6193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:01.173051 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "98983a658b734e82aca25dfe18964767"
format_stamp: "Formatted at 2026-08-08 04:45:01 on dist-test-slave-2kcd"
server_key: "724f9da09f8551e8921c15ee7c1e0248"
server_key_iv: "7e29269f1201d4598ce6de693dd5a5cd"
server_key_version: "encryptionkey@0"
I20260808 04:45:01.173600 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "98983a658b734e82aca25dfe18964767"
format_stamp: "Formatted at 2026-08-08 04:45:01 on dist-test-slave-2kcd"
server_key: "724f9da09f8551e8921c15ee7c1e0248"
server_key_iv: "7e29269f1201d4598ce6de693dd5a5cd"
server_key_version: "encryptionkey@0"
I20260808 04:45:01.177323 6193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:45:01.179708 6215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:01.180996 6193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:01.181097 6193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "98983a658b734e82aca25dfe18964767"
format_stamp: "Formatted at 2026-08-08 04:45:01 on dist-test-slave-2kcd"
server_key: "724f9da09f8551e8921c15ee7c1e0248"
server_key_iv: "7e29269f1201d4598ce6de693dd5a5cd"
server_key_version: "encryptionkey@0"
I20260808 04:45:01.181181 6193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:01.195021 6193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:01.197753 6193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:01.197902 6193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:01.198429 6193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:01.199218 6193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:01.199265 6193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:01.199304 6193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:01.199328 6193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:01.209460 6193 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:38067
I20260808 04:45:01.209563 6391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:38067 every 8 connection(s)
I20260808 04:45:01.210418 6193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[5803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164301, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:01.220624 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6193
I20260808 04:45:01.220762 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:01.221055 4964 external_mini_cluster.cc:1428] Setting key 5865b78ab5af7bc2b8363fc456342862
I20260808 04:45:01.224642 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.212223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:42727 (local address 127.4.217.62:38441)
0808 04:45:01.212372 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.212377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.213303 (+ 926us) server_negotiation.cc:407] Connection header received
0808 04:45:01.214746 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.214749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.214828 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.214924 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:01.216248 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.216838 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.217959 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.218177 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.218254 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.220731 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:01.220737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:01.220742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:01.220774 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:01.222443 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.222950 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.222957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.222961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.223058 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.223387 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.223394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.223398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.223665 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:01.223776 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.224237 (+ 461us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.224395 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:45:01.225523 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.212641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38441 (local address 127.4.217.2:42727)
0808 04:45:01.213157 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:01.213200 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:01.214534 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:01.215136 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:01.215147 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:01.215546 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:01.216078 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.216091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.216949 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.216952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.217852 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.217862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.218305 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.218308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.218499 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.219016 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:01.219038 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:01.220524 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:01.222566 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:01.222571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:01.222582 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:01.222838 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:01.223156 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:01.223159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:01.223160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:01.223292 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:01.223773 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:01.223777 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:01.223991 (+ 214us) client_negotiation.cc:769] Sending connection context
0808 04:45:01.224150 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 04:45:01.224394 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":170,"threads_started":1}
I20260808 04:45:01.226697 6392 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38441
I20260808 04:45:01.226950 6392 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:01.227525 6392 heartbeater.cc:511] Master 127.4.217.62:38441 requested a full tablet report, sending...
I20260808 04:45:01.228806 5866 ts_manager.cc:194] Registered new tserver with Master: 98983a658b734e82aca25dfe18964767 (127.4.217.2:38067)
I20260808 04:45:01.229708 5866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:42727
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:45:01.249920 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38441
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:01.365849 6401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:01.366099 6401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:01.366148 6401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:01.370088 6401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:01.370139 6401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:01.370196 6401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:01.375914 6401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38441
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:01.377113 6401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:01.378149 6401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:01.388677 6414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:01.388831 6401 server_base.cc:1034] running on GCE node
W20260808 04:45:01.388675 6415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:01.388675 6419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:01.389362 6401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:01.390053 6401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:01.391250 6401 hybrid_clock.cc:648] HybridClock initialized: now 1786164301391231 us; error 36 us; skew 500 ppm
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[5803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164301, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:01.394330 6401 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:45:01.395241 6401 webserver.cc:460] Webserver started at http://127.4.217.3:35365/ using document root <none> and password file <none>
I20260808 04:45:01.395871 6401 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:01.395979 6401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:01.396245 6401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:01.398517 6401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "ac749f0f52e640e8a07d559cf2dfd42e"
format_stamp: "Formatted at 2026-08-08 04:45:01 on dist-test-slave-2kcd"
server_key: "03686a12236421ab6103bfa4a52de432"
server_key_iv: "103cdd79c76e59e8727a7bd2d41e747b"
server_key_version: "encryptionkey@0"
I20260808 04:45:01.399159 6401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "ac749f0f52e640e8a07d559cf2dfd42e"
format_stamp: "Formatted at 2026-08-08 04:45:01 on dist-test-slave-2kcd"
server_key: "03686a12236421ab6103bfa4a52de432"
server_key_iv: "103cdd79c76e59e8727a7bd2d41e747b"
server_key_version: "encryptionkey@0"
I20260808 04:45:01.403291 6401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:45:01.405790 6430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:01.407027 6401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:01.407187 6401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "ac749f0f52e640e8a07d559cf2dfd42e"
format_stamp: "Formatted at 2026-08-08 04:45:01 on dist-test-slave-2kcd"
server_key: "03686a12236421ab6103bfa4a52de432"
server_key_iv: "103cdd79c76e59e8727a7bd2d41e747b"
server_key_version: "encryptionkey@0"
I20260808 04:45:01.407305 6401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:01.425268 6401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:01.428328 6401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:01.428545 6401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:01.429284 6401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:01.430151 6401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:01.430198 6401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:01.430239 6401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:01.430260 6401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:01.440689 6401 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:32787
I20260808 04:45:01.440714 6624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:32787 every 8 connection(s)
I20260808 04:45:01.441669 6401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:01.446331 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6401
I20260808 04:45:01.446465 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:01.446764 4964 external_mini_cluster.cc:1428] Setting key 29424038094e0b814b29958e8f07ce18
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[5803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164301, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:01.459084 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.443636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:45125 (local address 127.4.217.62:38441)
0808 04:45:01.443829 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.443834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.444729 (+ 895us) server_negotiation.cc:407] Connection header received
0808 04:45:01.446166 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.446170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.446235 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.446341 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:01.447694 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.448221 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.449648 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.449870 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.449992 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.455641 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:01.455645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:01.455648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:01.455689 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:01.456982 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.457509 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.457513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.457514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.457591 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.457946 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.457949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.457950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.458142 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:01.458251 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.458780 (+ 529us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.458910 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:45:01.460090 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.444019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38441 (local address 127.4.217.3:45125)
0808 04:45:01.444560 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:01.444604 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:01.445916 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:01.446574 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:01.446581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:01.446992 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:01.447530 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.447545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.448352 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.448356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.449507 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.449518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.450032 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.450037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.450202 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.453863 (+ 3661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:01.453880 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:01.455415 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:01.457101 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:01.457108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:01.457123 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:01.457381 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:01.457687 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:01.457691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:01.457693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:01.457835 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:01.458265 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:01.458273 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:01.458528 (+ 255us) client_negotiation.cc:769] Sending connection context
0808 04:45:01.458718 (+ 190us) client_negotiation.cc:240] Negotiation successful
0808 04:45:01.459064 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":116,"threads_started":1}
I20260808 04:45:01.461387 6625 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38441
I20260808 04:45:01.461606 6625 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:01.462081 6625 heartbeater.cc:511] Master 127.4.217.62:38441 requested a full tablet report, sending...
I20260808 04:45:01.463066 5866 ts_manager.cc:194] Registered new tserver with Master: ac749f0f52e640e8a07d559cf2dfd42e (127.4.217.3:32787)
I20260808 04:45:01.463641 5866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:45125
I20260808 04:45:01.472674 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:45:01.483021 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.476219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48174 (local address 127.4.217.62:38441)
0808 04:45:01.476345 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.476349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.476505 (+ 156us) server_negotiation.cc:407] Connection header received
0808 04:45:01.476606 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.476608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.476646 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.476717 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:01.477428 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.477953 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.478742 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.478953 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.479034 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.480327 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:01.480330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:01.480333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:01.480355 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:01.481638 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.482013 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.482015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.482016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.482087 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.482313 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.482315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.482316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.482523 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:01.482627 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.482820 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.482909 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:45:01.487797 5866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:01.490926 5866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:01.505527 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.500056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:40271 (local address 127.0.0.1:37120)
0808 04:45:01.500742 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:01.500756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:01.500877 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:01.501399 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:01.501405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:01.501427 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:01.501581 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.501588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.502831 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.502834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.503985 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.503996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.504835 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.504839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.505080 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.505131 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 04:45:01.505248 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 04:45:01.505371 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":91,"thread_start_us":97,"threads_started":1}
I20260808 04:45:01.505575 6656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.500370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.4.217.1:40271)
0808 04:45:01.500863 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.500881 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.500907 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:45:01.500991 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.501007 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.501221 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.501409 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:01.501750 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.502673 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.504131 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.504684 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.504835 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.505056 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.505302 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.505403 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":80,"threads_started":1}
I20260808 04:45:01.506423 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.499646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:38067 (local address 127.0.0.1:49126)
0808 04:45:01.500001 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:01.500052 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:01.500182 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:01.501754 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:01.501758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:01.501776 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:01.501881 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.501891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.503531 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.503534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.504874 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.504886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.506061 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.506065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.506156 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.506182 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:45:01.506232 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 04:45:01.506297 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":60,"thread_start_us":73,"threads_started":1}
I20260808 04:45:01.506788 6657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.500531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.4.217.3:32787)
0808 04:45:01.501529 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.501552 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.501574 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:45:01.501650 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.501658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.501826 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.501977 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:01.502248 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.503363 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.505170 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.506043 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.506147 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.506281 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.506406 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.506491 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":160,"threads_started":1}
I20260808 04:45:01.507061 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.499811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.4.217.2:38067)
0808 04:45:01.500974 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.500989 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.501012 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:45:01.501105 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.501374 (+ 269us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.501594 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.502030 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:01.502225 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.503373 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.505023 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.505900 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.506623 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.506770 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.506820 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.506903 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":137,"threads_started":1}
I20260808 04:45:01.507296 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.500358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:32787 (local address 127.0.0.1:48566)
0808 04:45:01.501072 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:01.501084 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:01.501284 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:01.501969 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:01.501972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:01.501992 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:01.502125 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.502132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.503528 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.503541 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.505067 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.505076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.506162 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.506165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.506266 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.506290 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:45:01.506375 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:45:01.506451 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":38,"thread_start_us":84,"threads_started":1}
I20260808 04:45:01.509065 6074 tablet_service.cc:1467] Processing CreateTablet for tablet d66d9e44dfad462cbedb3f614d8187d0 (DEFAULT_TABLE table=test-table [id=0243665086734f858c9e3cb8e5a26970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:01.509511 6289 tablet_service.cc:1467] Processing CreateTablet for tablet d66d9e44dfad462cbedb3f614d8187d0 (DEFAULT_TABLE table=test-table [id=0243665086734f858c9e3cb8e5a26970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:01.510259 6074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66d9e44dfad462cbedb3f614d8187d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:01.510491 6289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66d9e44dfad462cbedb3f614d8187d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:01.510069 6517 tablet_service.cc:1467] Processing CreateTablet for tablet d66d9e44dfad462cbedb3f614d8187d0 (DEFAULT_TABLE table=test-table [id=0243665086734f858c9e3cb8e5a26970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:01.511050 6517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66d9e44dfad462cbedb3f614d8187d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:01.518126 6658 tablet_bootstrap.cc:492] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767: Bootstrap starting.
I20260808 04:45:01.518780 6660 tablet_bootstrap.cc:492] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09: Bootstrap starting.
I20260808 04:45:01.520519 6660 tablet_bootstrap.cc:654] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:01.520668 6658 tablet_bootstrap.cc:654] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:01.521330 6660 log.cc:824] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:01.521443 6658 log.cc:824] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:01.522042 6661 tablet_bootstrap.cc:492] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e: Bootstrap starting.
I20260808 04:45:01.523442 6660 tablet_bootstrap.cc:492] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09: No bootstrap required, opened a new log
I20260808 04:45:01.523651 6660 ts_tablet_manager.cc:1397] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:45:01.524101 6661 tablet_bootstrap.cc:654] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:01.525301 6661 log.cc:824] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:01.526242 6658 tablet_bootstrap.cc:492] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767: No bootstrap required, opened a new log
I20260808 04:45:01.526436 6658 ts_tablet_manager.cc:1397] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260808 04:45:01.526995 6661 tablet_bootstrap.cc:492] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e: No bootstrap required, opened a new log
I20260808 04:45:01.527192 6661 ts_tablet_manager.cc:1397] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 04:45:01.528204 6660 raft_consensus.cc:348] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } }
I20260808 04:45:01.528481 6660 raft_consensus.cc:374] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:01.528575 6660 raft_consensus.cc:729] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d7f0dab60444d98d0df5885b139e09, State: Initialized, Role: FOLLOWER
I20260808 04:45:01.529060 6660 consensus_queue.cc:260] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } }
I20260808 04:45:01.530115 6660 ts_tablet_manager.cc:1428] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260808 04:45:01.530606 6181 heartbeater.cc:503] Master 127.4.217.62:38441 was elected leader, sending a full tablet report...
I20260808 04:45:01.530614 6661 raft_consensus.cc:348] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } }
I20260808 04:45:01.530875 6661 raft_consensus.cc:374] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:01.530695 6658 raft_consensus.cc:348] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } }
I20260808 04:45:01.530936 6661 raft_consensus.cc:729] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac749f0f52e640e8a07d559cf2dfd42e, State: Initialized, Role: FOLLOWER
I20260808 04:45:01.530982 6658 raft_consensus.cc:374] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:01.531057 6658 raft_consensus.cc:729] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98983a658b734e82aca25dfe18964767, State: Initialized, Role: FOLLOWER
I20260808 04:45:01.531371 6661 consensus_queue.cc:260] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } }
I20260808 04:45:01.531442 6658 consensus_queue.cc:260] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } }
I20260808 04:45:01.532115 6392 heartbeater.cc:503] Master 127.4.217.62:38441 was elected leader, sending a full tablet report...
I20260808 04:45:01.532447 6658 ts_tablet_manager.cc:1428] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:45:01.532462 6661 ts_tablet_manager.cc:1428] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 04:45:01.532765 6625 heartbeater.cc:503] Master 127.4.217.62:38441 was elected leader, sending a full tablet report...
W20260808 04:45:01.692764 6626 tablet.cc:2369] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:45:01.711616 6394 tablet.cc:2369] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:01.718183 6672 raft_consensus.cc:482] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:01.718413 6672 raft_consensus.cc:504] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } }
I20260808 04:45:01.719695 6672 leader_election.cc:288] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98983a658b734e82aca25dfe18964767 (127.4.217.2:38067), 19d7f0dab60444d98d0df5885b139e09 (127.4.217.1:40271)
I20260808 04:45:01.723628 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.719953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:40271 (local address 127.4.217.3:49941)
0808 04:45:01.720079 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:01.720095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:01.720200 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:01.720478 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:01.720481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:01.720501 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:01.720625 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.720630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.721662 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.721665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.722483 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.722493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.723183 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.723187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.723305 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.723323 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:45:01.723414 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 04:45:01.723486 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":24}
I20260808 04:45:01.723628 6656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.720080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:49941 (local address 127.4.217.1:40271)
0808 04:45:01.720232 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.720236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.720260 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:45:01.720312 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.720315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.720358 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.720459 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:01.721033 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.721528 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.722616 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.723055 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.723177 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.723243 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.723434 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.723507 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:45:01.724206 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.720071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:55943 (local address 127.4.217.2:38067)
0808 04:45:01.720196 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.720201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.720728 (+ 527us) server_negotiation.cc:407] Connection header received
0808 04:45:01.720821 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.720824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.720870 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.720930 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:01.721245 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.721787 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.722908 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.723309 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.723429 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.723491 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.724026 (+ 535us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.724098 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 04:45:01.724254 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.719955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:38067 (local address 127.4.217.3:55943)
0808 04:45:01.720618 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:01.720630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:01.720715 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:01.720959 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:01.720962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:01.720977 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:01.721100 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.721105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.721908 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.721912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.722773 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:01.722783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.723436 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.723440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:01.723605 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.723625 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 04:45:01.724002 (+ 377us) client_negotiation.cc:240] Negotiation successful
0808 04:45:01.724078 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":303,"thread_start_us":135,"threads_started":1}
I20260808 04:45:01.724346 6108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66d9e44dfad462cbedb3f614d8187d0" candidate_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d7f0dab60444d98d0df5885b139e09" is_pre_election: true
I20260808 04:45:01.724704 6108 raft_consensus.cc:2454] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac749f0f52e640e8a07d559cf2dfd42e in term 0.
I20260808 04:45:01.724985 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66d9e44dfad462cbedb3f614d8187d0" candidate_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98983a658b734e82aca25dfe18964767" is_pre_election: true
I20260808 04:45:01.725307 6322 raft_consensus.cc:2454] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac749f0f52e640e8a07d559cf2dfd42e in term 0.
I20260808 04:45:01.725293 6431 leader_election.cc:302] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d7f0dab60444d98d0df5885b139e09, ac749f0f52e640e8a07d559cf2dfd42e; no voters:
I20260808 04:45:01.725615 6672 raft_consensus.cc:2781] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:01.725724 6672 raft_consensus.cc:482] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:01.725790 6672 raft_consensus.cc:3037] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:01.726902 6672 raft_consensus.cc:504] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } }
I20260808 04:45:01.727270 6672 leader_election.cc:288] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [CANDIDATE]: Term 1 election: Requested vote from peers 98983a658b734e82aca25dfe18964767 (127.4.217.2:38067), 19d7f0dab60444d98d0df5885b139e09 (127.4.217.1:40271)
I20260808 04:45:01.727662 6108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66d9e44dfad462cbedb3f614d8187d0" candidate_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d7f0dab60444d98d0df5885b139e09"
I20260808 04:45:01.727653 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66d9e44dfad462cbedb3f614d8187d0" candidate_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98983a658b734e82aca25dfe18964767"
I20260808 04:45:01.727785 6322 raft_consensus.cc:3037] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:01.727785 6108 raft_consensus.cc:3037] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:01.729127 6108 raft_consensus.cc:2454] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac749f0f52e640e8a07d559cf2dfd42e in term 1.
I20260808 04:45:01.729127 6322 raft_consensus.cc:2454] T d66d9e44dfad462cbedb3f614d8187d0 P 98983a658b734e82aca25dfe18964767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac749f0f52e640e8a07d559cf2dfd42e in term 1.
I20260808 04:45:01.729478 6436 leader_election.cc:302] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98983a658b734e82aca25dfe18964767, ac749f0f52e640e8a07d559cf2dfd42e; no voters:
I20260808 04:45:01.729722 6672 raft_consensus.cc:2781] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:01.729923 6672 raft_consensus.cc:686] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [term 1 LEADER]: Becoming Leader. State: Replica: ac749f0f52e640e8a07d559cf2dfd42e, State: Running, Role: LEADER
I20260808 04:45:01.730240 6672 consensus_queue.cc:237] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } }
I20260808 04:45:01.733566 5872 catalog_manager.cc:5495] T d66d9e44dfad462cbedb3f614d8187d0 P ac749f0f52e640e8a07d559cf2dfd42e reported cstate change: term changed from 0 to 1, leader changed from <none> to ac749f0f52e640e8a07d559cf2dfd42e (127.4.217.3). New cstate: current_term: 1 leader_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98983a658b734e82aca25dfe18964767" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac749f0f52e640e8a07d559cf2dfd42e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 32787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19d7f0dab60444d98d0df5885b139e09" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 40271 } health_report { overall_health: UNKNOWN } } }
W20260808 04:45:01.736573 6187 tablet.cc:2369] T d66d9e44dfad462cbedb3f614d8187d0 P 19d7f0dab60444d98d0df5885b139e09: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[5803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:45:01.799295 6656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.790446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.4.217.1:40271)
0808 04:45:01.790645 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.790649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.790750 (+ 101us) server_negotiation.cc:407] Connection header received
0808 04:45:01.790860 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.790862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.790908 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.790980 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:01.791643 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.792142 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.793033 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.793235 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.793313 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.796000 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:01.796012 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:01.796028 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:01.796059 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:01.797732 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.798147 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.798153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.798154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.798258 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.798492 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.798494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.798496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.798737 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:01.798892 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.799072 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.799169 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[5803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[5803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:45:01.827447 6656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.819747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.4.217.1:40271)
0808 04:45:01.819907 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.819911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.819964 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:45:01.820095 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.820102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.820165 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.820255 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:01.820926 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.821452 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.822219 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.822432 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.822510 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.824705 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:01.824708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:01.824710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:01.824732 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:01.825960 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.826334 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.826336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.826337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.826410 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.826655 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.826657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.826659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.826810 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:01.826911 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.827141 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.827235 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[5803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[5803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:45:01.852433 6656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:01.844977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.4.217.1:40271)
0808 04:45:01.845115 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:01.845119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:01.845285 (+ 166us) server_negotiation.cc:407] Connection header received
0808 04:45:01.845367 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:01.845369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:01.845407 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:01.845472 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:01.846096 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.846558 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.847315 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:01.847513 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:01.847586 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:01.849848 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:01.849851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:01.849853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:01.849873 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:01.851043 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.851402 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.851404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.851405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.851474 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:01.851707 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:01.851709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:01.851710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:01.851849 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:01.851942 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:01.852136 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 04:45:01.852245 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:45:01.853995 6046 tablet_service.cc:2908] Rejecting scan request for tablet d66d9e44dfad462cbedb3f614d8187d0: Uninitialized: clean time has not yet been initialized
I20260808 04:45:01.856002 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 5983
I20260808 04:45:01.872519 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6193
I20260808 04:45:01.881016 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6401
I20260808 04:45:01.892180 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 5822
2026-08-08T04:45:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2816 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[6694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[6694](info): set up 2 sockets
Aug 08 04:45:01 dist-test-slave-2kcd krb5kdc[6694](info): commencing operation
krb5kdc: starting...
W20260808 04:45:03.165343 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:03 dist-test-slave-2kcd krb5kdc[6694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:03.289746 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:37223
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:44931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:37223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:03.408512 6718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:03.408810 6718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:03.408852 6718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:03.412741 6718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:03.412801 6718 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:45:03.412818 6718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:03.412837 6718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:03.412851 6718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:03.412866 6718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:03.418059 6718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:44931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:37223
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:37223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:03.419102 6718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:03.420140 6718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:03.428900 6730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:03.428907 6725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:03.428959 6724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:03.429306 6718 server_base.cc:1034] running on GCE node
I20260808 04:45:03.429908 6718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:03.431010 6718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:03.432206 6718 hybrid_clock.cc:648] HybridClock initialized: now 1786164303432187 us; error 31 us; skew 500 ppm
Aug 08 04:45:03 dist-test-slave-2kcd krb5kdc[6694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:03.434964 6718 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:03.435880 6718 webserver.cc:460] Webserver started at http://127.4.217.62:39629/ using document root <none> and password file <none>
I20260808 04:45:03.436450 6718 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:03.436554 6718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:03.436830 6718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:03.438859 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "25461ec4d20b4e3284f8b758a6140f04"
format_stamp: "Formatted at 2026-08-08 04:45:03 on dist-test-slave-2kcd"
server_key: "e6840cb68f97c1384fe1dc3c00b9d697"
server_key_iv: "099f9060a6ba8f66482349e02e742bb0"
server_key_version: "encryptionkey@0"
I20260808 04:45:03.439432 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "25461ec4d20b4e3284f8b758a6140f04"
format_stamp: "Formatted at 2026-08-08 04:45:03 on dist-test-slave-2kcd"
server_key: "e6840cb68f97c1384fe1dc3c00b9d697"
server_key_iv: "099f9060a6ba8f66482349e02e742bb0"
server_key_version: "encryptionkey@0"
I20260808 04:45:03.443194 6718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:45:03.445583 6744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:03.446791 6718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:03.446936 6718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "25461ec4d20b4e3284f8b758a6140f04"
format_stamp: "Formatted at 2026-08-08 04:45:03 on dist-test-slave-2kcd"
server_key: "e6840cb68f97c1384fe1dc3c00b9d697"
server_key_iv: "099f9060a6ba8f66482349e02e742bb0"
server_key_version: "encryptionkey@0"
I20260808 04:45:03.447048 6718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:03.463814 6718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:03.466779 6718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:03.466998 6718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:03.475013 6718 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:37223
I20260808 04:45:03.475037 6842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:37223 every 8 connection(s)
I20260808 04:45:03.476027 6718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:03.478879 6844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:03.484419 6844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04: Bootstrap starting.
I20260808 04:45:03.485905 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6718
I20260808 04:45:03.486001 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:03.486266 4964 external_mini_cluster.cc:1428] Setting key ccae269ca5bdeb1265cbf6162a93fcbd
I20260808 04:45:03.486972 6844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:03.487906 6844 log.cc:824] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:03.490337 6844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04: No bootstrap required, opened a new log
I20260808 04:45:03.494660 6844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25461ec4d20b4e3284f8b758a6140f04" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37223 } }
I20260808 04:45:03.494971 6844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:03.495067 6844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25461ec4d20b4e3284f8b758a6140f04, State: Initialized, Role: FOLLOWER
I20260808 04:45:03.495489 6844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25461ec4d20b4e3284f8b758a6140f04" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37223 } }
Aug 08 04:45:03 dist-test-slave-2kcd krb5kdc[6694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:03.495613 6844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:03.495698 6844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:03.495819 6844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:03.496681 6844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25461ec4d20b4e3284f8b758a6140f04" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37223 } }
I20260808 04:45:03.497051 6844 leader_election.cc:302] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25461ec4d20b4e3284f8b758a6140f04; no voters:
I20260808 04:45:03.497400 6844 leader_election.cc:288] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:03.497473 6857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:03.497658 6857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [term 1 LEADER]: Becoming Leader. State: Replica: 25461ec4d20b4e3284f8b758a6140f04, State: Running, Role: LEADER
I20260808 04:45:03.498004 6857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25461ec4d20b4e3284f8b758a6140f04" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37223 } }
I20260808 04:45:03.498733 6844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:03.501183 6859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25461ec4d20b4e3284f8b758a6140f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25461ec4d20b4e3284f8b758a6140f04" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37223 } } }
I20260808 04:45:03.500900 6858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25461ec4d20b4e3284f8b758a6140f04. Latest consensus state: current_term: 1 leader_uuid: "25461ec4d20b4e3284f8b758a6140f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25461ec4d20b4e3284f8b758a6140f04" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37223 } } }
I20260808 04:45:03.501327 6859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:03.501333 6858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:03.501773 6876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:03.502310 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:03.487382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.4.217.62:37223)
0808 04:45:03.487805 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:03.487821 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:03.487857 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:45:03.488713 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:03.488741 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:03.489117 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:03.489391 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:03.490325 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.491334 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.492602 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.493144 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.493266 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:03.496267 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:03.496272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:03.496285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:03.496312 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:03.498363 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:03.498823 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:03.498831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:03.498838 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:03.499004 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:03.499337 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:03.499342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:03.499344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:03.499916 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:03.500137 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:03.500480 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 04:45:03.500846 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260808 04:45:03.505225 6876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:03.510236 6876 catalog_manager.cc:1347] Generated new cluster ID: b91e94ad66374a099379cc3cd8ae1264
I20260808 04:45:03.510305 6876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:03.521486 6876 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:03.522348 6876 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:03.532672 6876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04: Generated new TSK 0
I20260808 04:45:03.533324 6876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:45:03.563422 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37223
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:44931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:03.685781 6883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:03.686050 6883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:03.686091 6883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:03.690140 6883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:03.690191 6883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:03.690229 6883 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:45:03.690260 6883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:03.695325 6883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:44931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.217.62:37223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:03.696270 6883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:03.697320 6883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:03.707001 6895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:03.706964 6890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:03.706986 6891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:03.707185 6883 server_base.cc:1034] running on GCE node
I20260808 04:45:03.707681 6883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:03.708384 6883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:03.709549 6883 hybrid_clock.cc:648] HybridClock initialized: now 1786164303709530 us; error 30 us; skew 500 ppm
Aug 08 04:45:03 dist-test-slave-2kcd krb5kdc[6694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:03.712385 6883 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:45:03.713286 6883 webserver.cc:460] Webserver started at http://127.4.217.1:32775/ using document root <none> and password file <none>
I20260808 04:45:03.713806 6883 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:03.713864 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:03.714035 6883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:03.716032 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "389c86ef70184abe9ce69542eb732c0c"
format_stamp: "Formatted at 2026-08-08 04:45:03 on dist-test-slave-2kcd"
server_key: "9b27999806f2c607df5d9734baae6850"
server_key_iv: "7b8592f6e3d3490d7dd58cfa9d26277f"
server_key_version: "encryptionkey@0"
I20260808 04:45:03.716534 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "389c86ef70184abe9ce69542eb732c0c"
format_stamp: "Formatted at 2026-08-08 04:45:03 on dist-test-slave-2kcd"
server_key: "9b27999806f2c607df5d9734baae6850"
server_key_iv: "7b8592f6e3d3490d7dd58cfa9d26277f"
server_key_version: "encryptionkey@0"
I20260808 04:45:03.720279 6883 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:45:03.722612 6906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:03.723853 6883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:03.724011 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "389c86ef70184abe9ce69542eb732c0c"
format_stamp: "Formatted at 2026-08-08 04:45:03 on dist-test-slave-2kcd"
server_key: "9b27999806f2c607df5d9734baae6850"
server_key_iv: "7b8592f6e3d3490d7dd58cfa9d26277f"
server_key_version: "encryptionkey@0"
I20260808 04:45:03.724121 6883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:03.753512 6883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:03.757115 6883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:03.757339 6883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:03.758888 6883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:03.758968 6883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:03.759076 6883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:03.759128 6883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:45:03 dist-test-slave-2kcd krb5kdc[6694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:03.771602 6883 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:38943
I20260808 04:45:03.771703 7105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:38943 every 8 connection(s)
I20260808 04:45:03.772742 6883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:03.774199 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:03.759867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:37873 (local address 127.4.217.62:37223)
0808 04:45:03.760016 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:03.760023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:03.761185 (+ 1162us) server_negotiation.cc:407] Connection header received
0808 04:45:03.762216 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:03.762219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:03.762282 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:03.762370 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:03.763602 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.764462 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.765890 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.766140 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.766262 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:03.770229 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:03.770236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:03.770239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:03.770275 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:03.772000 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:03.772517 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:03.772521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:03.772523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:03.772620 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:03.773051 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:03.773055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:03.773057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:03.773323 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:03.773430 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:03.773952 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 04:45:03.774073 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:03.775022 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:03.760470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37223 (local address 127.4.217.1:37873)
0808 04:45:03.761033 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:03.761077 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:03.761999 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:03.762661 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:03.762667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:03.762979 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:03.763453 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:03.763467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.764623 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.764628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:03.765752 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:03.765763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.766306 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.766310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:03.766552 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:03.767191 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:03.767216 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:03.769994 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:03.772122 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:03.772127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:03.772137 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:03.772381 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:03.772718 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:03.772721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:03.772722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:03.772935 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:03.773424 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:03.773430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:03.773681 (+ 251us) client_negotiation.cc:769] Sending connection context
0808 04:45:03.773874 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 04:45:03.774177 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":151,"threads_started":1}
I20260808 04:45:03.776476 7106 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37223
I20260808 04:45:03.776650 7106 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:03.777179 7106 heartbeater.cc:511] Master 127.4.217.62:37223 requested a full tablet report, sending...
I20260808 04:45:03.778821 6774 ts_manager.cc:194] Registered new tserver with Master: 389c86ef70184abe9ce69542eb732c0c (127.4.217.1:38943)
I20260808 04:45:03.780299 6774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:37873
I20260808 04:45:03.780521 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6883
I20260808 04:45:03.780635 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:03.780975 4964 external_mini_cluster.cc:1428] Setting key b10db3b22cd8ec2df577bd1e9084427a
I20260808 04:45:03.786617 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:03.778722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37223 (local address 127.4.217.1:58567)
0808 04:45:03.779041 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:03.779056 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:03.779156 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:03.779406 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:03.779408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:03.779618 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:03.779745 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:03.779752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.780558 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.780561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:03.781407 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:03.781419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.781984 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.781988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:03.782078 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:03.782608 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:03.782623 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:03.783032 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:03.784978 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:03.784983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:03.784987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:03.785283 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:03.785555 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:03.785560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:03.785564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:03.785671 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:03.786026 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:03.786041 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:03.786151 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 04:45:03.786277 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 04:45:03.786419 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":85,"threads_started":1}
I20260808 04:45:03.786728 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:03.778797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:58567 (local address 127.4.217.62:37223)
0808 04:45:03.778911 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:03.778933 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:03.779128 (+ 195us) server_negotiation.cc:407] Connection header received
0808 04:45:03.779232 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:03.779234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:03.779292 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:03.779358 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:03.779860 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.780364 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.781544 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.781855 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.781957 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:03.783188 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:03.783194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:03.783199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:03.783231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:03.784859 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:03.785389 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:03.785391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:03.785392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:03.785467 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:03.785768 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:03.785770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:03.785771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:03.785925 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:03.786007 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:03.786355 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 04:45:03.786492 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:45:03.809850 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37223
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:44931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:03.923415 7122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:03.923651 7122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:03.923694 7122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:03.927651 7122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:03.927708 7122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:03.927747 7122 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:45:03.927819 7122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:03.932912 7122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:44931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.217.62:37223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:03.933818 7122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:03.934772 7122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:03.944362 7133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:03.944432 7132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:03.944625 7122 server_base.cc:1034] running on GCE node
W20260808 04:45:03.944387 7136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:03.945250 7122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:03.945828 7122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:03.946990 7122 hybrid_clock.cc:648] HybridClock initialized: now 1786164303946969 us; error 28 us; skew 500 ppm
Aug 08 04:45:03 dist-test-slave-2kcd krb5kdc[6694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:03.949839 7122 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:45:03.950717 7122 webserver.cc:460] Webserver started at http://127.4.217.2:46511/ using document root <none> and password file <none>
I20260808 04:45:03.951277 7122 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:03.951366 7122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:03.951601 7122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:03.953723 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "4819b5e2593f4296a09a5ab7e5fc52df"
format_stamp: "Formatted at 2026-08-08 04:45:03 on dist-test-slave-2kcd"
server_key: "5a61f5f163b2901311d4a9aa77f4ea79"
server_key_iv: "b5659ef77126d959f9dca4342e6cac3a"
server_key_version: "encryptionkey@0"
I20260808 04:45:03.954290 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "4819b5e2593f4296a09a5ab7e5fc52df"
format_stamp: "Formatted at 2026-08-08 04:45:03 on dist-test-slave-2kcd"
server_key: "5a61f5f163b2901311d4a9aa77f4ea79"
server_key_iv: "b5659ef77126d959f9dca4342e6cac3a"
server_key_version: "encryptionkey@0"
I20260808 04:45:03.958009 7122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:03.960472 7151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:03.961746 7122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:03.961896 7122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "4819b5e2593f4296a09a5ab7e5fc52df"
format_stamp: "Formatted at 2026-08-08 04:45:03 on dist-test-slave-2kcd"
server_key: "5a61f5f163b2901311d4a9aa77f4ea79"
server_key_iv: "b5659ef77126d959f9dca4342e6cac3a"
server_key_version: "encryptionkey@0"
I20260808 04:45:03.962007 7122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:03.984622 7122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:03.988125 7122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:03.988341 7122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:03.989972 7122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:03.990052 7122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:03.990132 7122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:03.990175 7122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:45:04 dist-test-slave-2kcd krb5kdc[6694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:04.002171 7122 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:43761
I20260808 04:45:04.002290 7330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:43761 every 8 connection(s)
I20260808 04:45:04.003307 7122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:04.006004 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:03.990886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:42067 (local address 127.4.217.62:37223)
0808 04:45:03.991024 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:03.991028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:03.992120 (+ 1092us) server_negotiation.cc:407] Connection header received
0808 04:45:03.993244 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:03.993248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:03.993316 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:03.993406 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:03.995236 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.996142 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.997604 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.998008 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.998157 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.002264 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:04.002267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:04.002270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:04.002302 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:04.003872 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.004509 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.004512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.004513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.004600 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.005003 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.005006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.005007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.005207 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:04.005289 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.005766 (+ 477us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.005878 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:45:04.006100 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 7122
I20260808 04:45:04.006201 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:04.006479 4964 external_mini_cluster.cc:1428] Setting key 704bdfdb4998ba393bfe83805ddec053
I20260808 04:45:04.007041 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:03.991456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37223 (local address 127.4.217.2:42067)
0808 04:45:03.991953 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:03.992009 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:03.992986 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:03.993778 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:03.993788 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:03.994304 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:03.994976 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:03.995026 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.996318 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.996323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:03.997459 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:03.997470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:03.998251 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:03.998257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:03.998485 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:03.999860 (+ 1375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:03.999884 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:04.002045 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:04.004020 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.004026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.004042 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.004391 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.004694 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.004696 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.004697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.004895 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.005302 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:04.005307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:04.005545 (+ 238us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.005697 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.006047 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":146,"threads_started":1}
I20260808 04:45:04.008513 7331 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37223
I20260808 04:45:04.008695 7331 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:04.009310 7331 heartbeater.cc:511] Master 127.4.217.62:37223 requested a full tablet report, sending...
I20260808 04:45:04.010517 6774 ts_manager.cc:194] Registered new tserver with Master: 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761)
I20260808 04:45:04.011106 6774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:42067
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:45:04.020634 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.011616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37223 (local address 127.4.217.2:58521)
0808 04:45:04.011909 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.011926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.012022 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.013175 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.013181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.013448 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:04.013629 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.013640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.014407 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.014412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.015450 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.015464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.015955 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.015959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.016086 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.016617 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:04.016636 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:04.017128 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:04.018870 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.018873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.018874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.019127 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.019437 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.019439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.019441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.019534 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.020060 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:04.020064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:04.020169 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.020283 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.020423 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260808 04:45:04.020663 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.011678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:58521 (local address 127.4.217.62:37223)
0808 04:45:04.012926 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.012930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.012945 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:04.012979 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.012981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.013025 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.013119 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:04.013753 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.014236 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.015598 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.015828 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.015912 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.017267 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:04.017270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:04.017272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:04.017294 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:04.018762 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.019253 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.019256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.019257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.019336 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.019639 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.019642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.019643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.019891 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:04.020069 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.020376 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.020526 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1164}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:45:04.038132 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37223
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:44931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:04.151819 7348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:04.152093 7348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:04.152166 7348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:04.156080 7348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:04.156131 7348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:04.156165 7348 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:45:04.156194 7348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:04.161283 7348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:44931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.217.62:37223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:04.162240 7348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:04.163195 7348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:04.172590 7359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:04.172693 7358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:04.172652 7361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:04.172843 7348 server_base.cc:1034] running on GCE node
I20260808 04:45:04.173265 7348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:04.173823 7348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:04.174981 7348 hybrid_clock.cc:648] HybridClock initialized: now 1786164304174964 us; error 32 us; skew 500 ppm
Aug 08 04:45:04 dist-test-slave-2kcd krb5kdc[6694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:04.177838 7348 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:45:04.178684 7348 webserver.cc:460] Webserver started at http://127.4.217.3:33741/ using document root <none> and password file <none>
I20260808 04:45:04.179214 7348 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:04.179306 7348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:04.179538 7348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:04.181509 7348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "b873269318a54ee3afc18e78e628e13b"
format_stamp: "Formatted at 2026-08-08 04:45:04 on dist-test-slave-2kcd"
server_key: "bc6612f2d01d9e8b166a4198144bbc58"
server_key_iv: "42ab5d01cfd61b2ea40c480d5cf61d94"
server_key_version: "encryptionkey@0"
I20260808 04:45:04.182071 7348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "b873269318a54ee3afc18e78e628e13b"
format_stamp: "Formatted at 2026-08-08 04:45:04 on dist-test-slave-2kcd"
server_key: "bc6612f2d01d9e8b166a4198144bbc58"
server_key_iv: "42ab5d01cfd61b2ea40c480d5cf61d94"
server_key_version: "encryptionkey@0"
I20260808 04:45:04.185772 7348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:04.188097 7374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:04.189278 7348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:04.189424 7348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "b873269318a54ee3afc18e78e628e13b"
format_stamp: "Formatted at 2026-08-08 04:45:04 on dist-test-slave-2kcd"
server_key: "bc6612f2d01d9e8b166a4198144bbc58"
server_key_iv: "42ab5d01cfd61b2ea40c480d5cf61d94"
server_key_version: "encryptionkey@0"
I20260808 04:45:04.189543 7348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:04.207126 7348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:04.210070 7348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:04.210268 7348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:04.211733 7348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:04.211808 7348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:04.211900 7348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:04.211947 7348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:45:04 dist-test-slave-2kcd krb5kdc[6694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:04.223366 7348 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:41429
I20260808 04:45:04.223451 7565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:41429 every 8 connection(s)
I20260808 04:45:04.224609 7348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:04.226972 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.212401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:57267 (local address 127.4.217.62:37223)
0808 04:45:04.212574 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.212580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.213967 (+ 1387us) server_negotiation.cc:407] Connection header received
0808 04:45:04.215483 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.215487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.215553 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.215653 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:04.217336 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.217956 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.219826 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.220120 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.220206 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.223257 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:04.223263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:04.223268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:04.223303 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:04.224894 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.225493 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.225497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.225499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.225597 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.225973 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.225975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.225976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.226171 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:04.226300 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.226748 (+ 448us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.226849 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:45:04.227795 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.213328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37223 (local address 127.4.217.3:57267)
0808 04:45:04.213810 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.213864 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.215200 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.215859 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.215872 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.216391 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:04.217142 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.217166 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.218098 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.218104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.219685 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.219700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.220298 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.220304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.220541 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.221179 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:04.221206 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:04.223080 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:04.225039 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.225046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.225060 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.225380 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.225685 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.225689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.225692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.225869 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.226299 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:04.226304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:04.226524 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.226669 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.226896 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260808 04:45:04.229040 7566 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37223
I20260808 04:45:04.229182 7566 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:04.229736 7566 heartbeater.cc:511] Master 127.4.217.62:37223 requested a full tablet report, sending...
I20260808 04:45:04.230857 6780 ts_manager.cc:194] Registered new tserver with Master: b873269318a54ee3afc18e78e628e13b (127.4.217.3:41429)
I20260808 04:45:04.231525 6780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:57267
I20260808 04:45:04.234694 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 7348
I20260808 04:45:04.234818 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:04.235097 4964 external_mini_cluster.cc:1428] Setting key 964c38d8fa37b4a13c406bb23e619672
I20260808 04:45:04.237341 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:45:04.238754 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.231449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:34897 (local address 127.4.217.62:37223)
0808 04:45:04.231584 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.231589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.231833 (+ 244us) server_negotiation.cc:407] Connection header received
0808 04:45:04.231965 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.231968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.232012 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.232100 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:04.232581 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.233229 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.234064 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.234280 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.234387 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.235438 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:04.235441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:04.235444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:04.235467 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:04.237093 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.237599 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.237603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.237605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.237691 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.237973 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.237977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.237979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.238144 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:04.238249 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.238507 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.238618 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:45:04.238768 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.231389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37223 (local address 127.4.217.3:34897)
0808 04:45:04.231734 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.231749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.231858 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.232118 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.232138 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.232323 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:04.232458 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.232464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.233382 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.233384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.233933 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.233942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.234394 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.234398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.234495 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.234962 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:04.234973 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:04.235308 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:04.237222 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.237226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.237228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.237491 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.237786 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.237790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.237792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.237880 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.238245 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:04.238248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:04.238344 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.238453 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.238589 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":3,"thread_start_us":133,"threads_started":1}
Aug 08 04:45:04 dist-test-slave-2kcd krb5kdc[6694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:45:04 dist-test-slave-2kcd krb5kdc[6694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:04.267619 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.259842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.4.217.62:37223)
0808 04:45:04.259960 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.259964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.259985 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:45:04.260022 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.260024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.260061 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.260125 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:04.260749 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.261281 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.262044 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.262281 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.262357 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.264627 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:04.264630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:04.264632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:04.264655 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:04.266232 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.266603 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.266605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.266606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.266690 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.266937 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.266940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.266941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.267098 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:04.267199 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.267399 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.267489 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:45:04.270560 6780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:04.273597 6780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:04.288391 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.283072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.4.217.2:43761)
0808 04:45:04.283524 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.283549 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.283572 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:45:04.283653 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.283658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.283855 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.284020 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:04.284241 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.285098 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.286738 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.287589 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.287704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.287874 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.288148 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.288243 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":117,"threads_started":1}
I20260808 04:45:04.288380 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.282045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:43761 (local address 127.0.0.1:33958)
0808 04:45:04.282611 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.282624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.282712 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.283999 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.284002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.284021 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:04.284124 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.284130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.285222 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.285225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.286605 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.286617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.287730 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.287734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.287864 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.287971 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.288102 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.288219 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":100,"thread_start_us":79,"threads_started":1}
I20260808 04:45:04.288720 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.282100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.4.217.3:41429)
0808 04:45:04.282721 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.282738 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.282784 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:45:04.282868 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.282874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.283118 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.283666 (+ 548us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:04.283801 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.285047 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.286767 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.287589 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.287704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.287871 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.288444 (+ 573us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.288543 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":126,"threads_started":1}
I20260808 04:45:04.288398 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.282628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:38943 (local address 127.0.0.1:42186)
0808 04:45:04.282881 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.282893 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.282971 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.284137 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.284140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.284159 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:04.284252 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.284257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.285907 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.285911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.287053 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.287063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.287876 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.287879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.287983 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.288005 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.288057 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.288208 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":18,"thread_start_us":110,"threads_started":1}
I20260808 04:45:04.288987 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.283277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.4.217.1:38943)
0808 04:45:04.283691 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.283697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.283718 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:45:04.283795 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.283800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.283980 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.284391 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:04.284545 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.285775 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.287177 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.287744 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.288471 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.288650 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.288700 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.288827 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":97,"threads_started":1}
I20260808 04:45:04.288398 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.281937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:41429 (local address 127.0.0.1:33110)
0808 04:45:04.282405 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.282457 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.282603 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.283323 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.283330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.283355 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:04.283494 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.283502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.285166 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.285172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.286620 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.286629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.287920 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.287923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.288015 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.288041 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.288108 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.288207 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":75,"thread_start_us":79,"threads_started":1}
I20260808 04:45:04.291620 7230 tablet_service.cc:1467] Processing CreateTablet for tablet 46b5f987da934b4798256604614cb75f (DEFAULT_TABLE table=test-table [id=be28dbd7bc6745b9817a6636a4b67873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:04.291797 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 46b5f987da934b4798256604614cb75f (DEFAULT_TABLE table=test-table [id=be28dbd7bc6745b9817a6636a4b67873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:04.291795 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 46b5f987da934b4798256604614cb75f (DEFAULT_TABLE table=test-table [id=be28dbd7bc6745b9817a6636a4b67873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:04.292644 7230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b5f987da934b4798256604614cb75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:04.292834 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b5f987da934b4798256604614cb75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:04.292845 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b5f987da934b4798256604614cb75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:04.300016 7600 tablet_bootstrap.cc:492] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: Bootstrap starting.
I20260808 04:45:04.301187 7602 tablet_bootstrap.cc:492] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: Bootstrap starting.
I20260808 04:45:04.301561 7603 tablet_bootstrap.cc:492] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: Bootstrap starting.
I20260808 04:45:04.302605 7600 tablet_bootstrap.cc:654] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:04.303052 7602 tablet_bootstrap.cc:654] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:04.303524 7600 log.cc:824] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:04.303645 7603 tablet_bootstrap.cc:654] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:04.303800 7602 log.cc:824] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:04.304457 7603 log.cc:824] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:04.305771 7602 tablet_bootstrap.cc:492] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: No bootstrap required, opened a new log
I20260808 04:45:04.306008 7602 ts_tablet_manager.cc:1397] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:45:04.306247 7600 tablet_bootstrap.cc:492] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: No bootstrap required, opened a new log
I20260808 04:45:04.306246 7603 tablet_bootstrap.cc:492] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: No bootstrap required, opened a new log
I20260808 04:45:04.306437 7600 ts_tablet_manager.cc:1397] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:45:04.306437 7603 ts_tablet_manager.cc:1397] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 04:45:04.309607 7602 raft_consensus.cc:348] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } }
I20260808 04:45:04.309844 7602 raft_consensus.cc:374] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:04.309899 7602 raft_consensus.cc:729] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389c86ef70184abe9ce69542eb732c0c, State: Initialized, Role: FOLLOWER
I20260808 04:45:04.310349 7602 consensus_queue.cc:260] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } }
I20260808 04:45:04.310451 7600 raft_consensus.cc:348] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } }
I20260808 04:45:04.310701 7600 raft_consensus.cc:374] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:04.310787 7600 raft_consensus.cc:729] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4819b5e2593f4296a09a5ab7e5fc52df, State: Initialized, Role: FOLLOWER
I20260808 04:45:04.311182 7106 heartbeater.cc:503] Master 127.4.217.62:37223 was elected leader, sending a full tablet report...
I20260808 04:45:04.311228 7600 consensus_queue.cc:260] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } }
I20260808 04:45:04.311291 7602 ts_tablet_manager.cc:1428] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 04:45:04.311893 7331 heartbeater.cc:503] Master 127.4.217.62:37223 was elected leader, sending a full tablet report...
I20260808 04:45:04.312207 7600 ts_tablet_manager.cc:1428] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:45:04.310451 7603 raft_consensus.cc:348] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } }
I20260808 04:45:04.312763 7603 raft_consensus.cc:374] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:04.312856 7603 raft_consensus.cc:729] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b873269318a54ee3afc18e78e628e13b, State: Initialized, Role: FOLLOWER
I20260808 04:45:04.313246 7603 consensus_queue.cc:260] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } }
I20260808 04:45:04.314162 7603 ts_tablet_manager.cc:1428] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:45:04.314553 7566 heartbeater.cc:503] Master 127.4.217.62:37223 was elected leader, sending a full tablet report...
I20260808 04:45:04.400050 7613 raft_consensus.cc:482] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:04.400327 7613 raft_consensus.cc:504] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } }
I20260808 04:45:04.401690 7613 leader_election.cc:288] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761), 389c86ef70184abe9ce69542eb732c0c (127.4.217.1:38943)
I20260808 04:45:04.406296 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.402047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:43761 (local address 127.4.217.3:45651)
0808 04:45:04.402189 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.402208 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.402311 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.402802 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.402805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.402830 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:04.402955 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.402961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.404018 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.404021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.405287 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.405299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.405955 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.405958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.406053 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.406074 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.406135 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.406192 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 04:45:04.406371 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.402118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:45651 (local address 127.4.217.2:43761)
0808 04:45:04.402254 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.402259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.402275 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:45:04.402441 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.402448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.402516 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.402602 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:04.403126 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.403903 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.405422 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.405842 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.405948 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.406034 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.406174 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.406249 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:04.407027 7263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b5f987da934b4798256604614cb75f" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" is_pre_election: true
I20260808 04:45:04.407155 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.402635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:50903 (local address 127.4.217.1:38943)
0808 04:45:04.402797 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.402802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.403017 (+ 215us) server_negotiation.cc:407] Connection header received
0808 04:45:04.403123 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.403126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.403179 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.403260 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:04.403554 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.404186 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.405885 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.406490 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.406616 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.406675 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.407004 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.407062 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:45:04.407186 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.402347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:38943 (local address 127.4.217.3:50903)
0808 04:45:04.402902 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.402918 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.403029 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.403285 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.403287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.403303 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:04.403435 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.403439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.404291 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.404294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.405759 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.405768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.406748 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.406752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.406856 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.406896 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.406980 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.407050 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":175,"threads_started":1}
I20260808 04:45:04.407380 7263 raft_consensus.cc:2454] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 0.
I20260808 04:45:04.407819 7031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b5f987da934b4798256604614cb75f" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389c86ef70184abe9ce69542eb732c0c" is_pre_election: true
I20260808 04:45:04.407897 7383 leader_election.cc:302] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4819b5e2593f4296a09a5ab7e5fc52df, b873269318a54ee3afc18e78e628e13b; no voters:
I20260808 04:45:04.408070 7031 raft_consensus.cc:2454] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 0.
I20260808 04:45:04.408239 7613 raft_consensus.cc:2781] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:04.408360 7613 raft_consensus.cc:482] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:04.408411 7613 raft_consensus.cc:3037] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:04.409672 7613 raft_consensus.cc:504] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } }
I20260808 04:45:04.410019 7613 leader_election.cc:288] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 election: Requested vote from peers 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761), 389c86ef70184abe9ce69542eb732c0c (127.4.217.1:38943)
I20260808 04:45:04.410392 7263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b5f987da934b4798256604614cb75f" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4819b5e2593f4296a09a5ab7e5fc52df"
I20260808 04:45:04.410482 7031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b5f987da934b4798256604614cb75f" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389c86ef70184abe9ce69542eb732c0c"
I20260808 04:45:04.410544 7263 raft_consensus.cc:3037] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:04.410640 7031 raft_consensus.cc:3037] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:04.411664 7031 raft_consensus.cc:2454] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 1.
I20260808 04:45:04.411964 7383 leader_election.cc:302] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 389c86ef70184abe9ce69542eb732c0c, b873269318a54ee3afc18e78e628e13b; no voters:
I20260808 04:45:04.412127 7613 raft_consensus.cc:2781] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:04.412287 7263 raft_consensus.cc:2454] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 1.
I20260808 04:45:04.412393 7613 raft_consensus.cc:686] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 1 LEADER]: Becoming Leader. State: Replica: b873269318a54ee3afc18e78e628e13b, State: Running, Role: LEADER
I20260808 04:45:04.412741 7613 consensus_queue.cc:237] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } }
I20260808 04:45:04.415993 6772 catalog_manager.cc:5495] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b reported cstate change: term changed from 0 to 1, leader changed from <none> to b873269318a54ee3afc18e78e628e13b (127.4.217.3). New cstate: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:04.445892 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.442386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.4.217.3:41429)
0808 04:45:04.442535 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.442538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.442554 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:45:04.442754 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.442756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.442818 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.442907 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:04.443152 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.443647 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.444475 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.444691 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.444767 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.445131 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:04.445536 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:04.445626 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.445718 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.445772 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:45:04.452105 7263 raft_consensus.cc:1261] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Refusing update from remote peer b873269318a54ee3afc18e78e628e13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:04.452105 7031 raft_consensus.cc:1261] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Refusing update from remote peer b873269318a54ee3afc18e78e628e13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:04.452883 7613 consensus_queue.cc:1035] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:04.453096 7616 consensus_queue.cc:1035] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:04.461114 7625 mvcc.cc:205] Tried to move back new op lower bound from 7316128991028109312 to 7316128990877368320. Current Snapshot: MvccSnapshot[applied={T|T < 7316128991028109312}]
I20260808 04:45:04.474642 6772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:04.474934 6772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:04.477730 6772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04: Sending DeleteTablet for 3 replicas of tablet 46b5f987da934b4798256604614cb75f
I20260808 04:45:04.478557 7463 tablet_service.cc:1514] Processing DeleteTablet for tablet 46b5f987da934b4798256604614cb75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:04 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:33110
I20260808 04:45:04.478564 7230 tablet_service.cc:1514] Processing DeleteTablet for tablet 46b5f987da934b4798256604614cb75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:04 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:33958
I20260808 04:45:04.479278 7636 tablet_replica.cc:330] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: stopping tablet replica
I20260808 04:45:04.479569 7635 tablet_replica.cc:330] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: stopping tablet replica
I20260808 04:45:04.479534 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet 46b5f987da934b4798256604614cb75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:04 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:42186
I20260808 04:45:04.480013 7635 raft_consensus.cc:2229] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:04.480352 7636 raft_consensus.cc:2229] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:04.481088 7635 raft_consensus.cc:2258] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:04.481279 7636 raft_consensus.cc:2258] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:04.481206 6774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:04.481666 6774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:04.483516 7636 ts_tablet_manager.cc:1905] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:04.485700 7635 ts_tablet_manager.cc:1905] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:04.486915 7636 ts_tablet_manager.cc:1918] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:04.487105 7636 log.cc:1196] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/46b5f987da934b4798256604614cb75f
I20260808 04:45:04.487643 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf0b403bb6740f0b0bbc259f3345e03 (DEFAULT_TABLE table=test-table [id=f12399a3c37b48f799eca1634635ca91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:04.487953 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf0b403bb6740f0b0bbc259f3345e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:04.487957 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf0b403bb6740f0b0bbc259f3345e03 (DEFAULT_TABLE table=test-table [id=f12399a3c37b48f799eca1634635ca91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:04.488225 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf0b403bb6740f0b0bbc259f3345e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:04.488615 7635 ts_tablet_manager.cc:1918] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:04.488711 7635 log.cc:1196] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/46b5f987da934b4798256604614cb75f
I20260808 04:45:04.489107 7635 ts_tablet_manager.cc:1939] T 46b5f987da934b4798256604614cb75f P 4819b5e2593f4296a09a5ab7e5fc52df: Deleting consensus metadata
I20260808 04:45:04.490424 7603 tablet_bootstrap.cc:492] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: Bootstrap starting.
I20260808 04:45:04.490420 6753 catalog_manager.cc:4841] TS 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761): tablet 46b5f987da934b4798256604614cb75f (table test-table [id=be28dbd7bc6745b9817a6636a4b67873]) successfully deleted
I20260808 04:45:04.491183 7230 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf0b403bb6740f0b0bbc259f3345e03 (DEFAULT_TABLE table=test-table [id=f12399a3c37b48f799eca1634635ca91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:04.491438 7230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf0b403bb6740f0b0bbc259f3345e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:04.491600 7603 tablet_bootstrap.cc:654] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:04.492096 7638 tablet_replica.cc:330] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: stopping tablet replica
I20260808 04:45:04.492431 7638 raft_consensus.cc:2229] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:04.492758 7638 raft_consensus.cc:2258] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:04.493150 7636 ts_tablet_manager.cc:1939] T 46b5f987da934b4798256604614cb75f P b873269318a54ee3afc18e78e628e13b: Deleting consensus metadata
I20260808 04:45:04.493279 7602 tablet_bootstrap.cc:492] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: Bootstrap starting.
I20260808 04:45:04.493994 7600 tablet_bootstrap.cc:492] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: Bootstrap starting.
I20260808 04:45:04.494356 7602 tablet_bootstrap.cc:654] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:04.494716 6747 catalog_manager.cc:4841] TS b873269318a54ee3afc18e78e628e13b (127.4.217.3:41429): tablet 46b5f987da934b4798256604614cb75f (table test-table [id=be28dbd7bc6745b9817a6636a4b67873]) successfully deleted
I20260808 04:45:04.495015 7600 tablet_bootstrap.cc:654] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:04.496037 7602 tablet_bootstrap.cc:492] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: No bootstrap required, opened a new log
I20260808 04:45:04.496129 7602 ts_tablet_manager.cc:1397] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:04.496292 7600 tablet_bootstrap.cc:492] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: No bootstrap required, opened a new log
I20260808 04:45:04.496362 7600 ts_tablet_manager.cc:1397] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:04.496665 7602 raft_consensus.cc:348] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.496774 7602 raft_consensus.cc:374] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:04.496850 7602 raft_consensus.cc:729] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389c86ef70184abe9ce69542eb732c0c, State: Initialized, Role: FOLLOWER
I20260808 04:45:04.496899 7600 raft_consensus.cc:348] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.497006 7600 raft_consensus.cc:374] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:04.497007 7638 ts_tablet_manager.cc:1905] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:04.497045 7600 raft_consensus.cc:729] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4819b5e2593f4296a09a5ab7e5fc52df, State: Initialized, Role: FOLLOWER
I20260808 04:45:04.497179 7600 consensus_queue.cc:260] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.497412 7603 tablet_bootstrap.cc:492] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: No bootstrap required, opened a new log
I20260808 04:45:04.497438 7600 ts_tablet_manager.cc:1428] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:04.497494 7603 ts_tablet_manager.cc:1397] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260808 04:45:04.497920 7603 raft_consensus.cc:348] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.498039 7603 raft_consensus.cc:374] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:04.498078 7603 raft_consensus.cc:729] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b873269318a54ee3afc18e78e628e13b, State: Initialized, Role: FOLLOWER
I20260808 04:45:04.498207 7603 consensus_queue.cc:260] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.498481 7603 ts_tablet_manager.cc:1428] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:45:04.498908 7602 consensus_queue.cc:260] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.499722 7602 ts_tablet_manager.cc:1428] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260808 04:45:04.499859 7638 ts_tablet_manager.cc:1918] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:04.500010 7638 log.cc:1196] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/46b5f987da934b4798256604614cb75f
I20260808 04:45:04.500396 7638 ts_tablet_manager.cc:1939] T 46b5f987da934b4798256604614cb75f P 389c86ef70184abe9ce69542eb732c0c: Deleting consensus metadata
I20260808 04:45:04.501817 6753 catalog_manager.cc:4841] TS 389c86ef70184abe9ce69542eb732c0c (127.4.217.1:38943): tablet 46b5f987da934b4798256604614cb75f (table test-table [id=be28dbd7bc6745b9817a6636a4b67873]) successfully deleted
W20260808 04:45:04.504761 7332 tablet.cc:2369] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:45:04.524212 7107 tablet.cc:2369] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:04.584679 7613 raft_consensus.cc:482] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:04.584930 7613 raft_consensus.cc:504] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.585413 7613 leader_election.cc:288] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 389c86ef70184abe9ce69542eb732c0c (127.4.217.1:38943), 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761)
I20260808 04:45:04.585865 7031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf0b403bb6740f0b0bbc259f3345e03" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389c86ef70184abe9ce69542eb732c0c" is_pre_election: true
I20260808 04:45:04.586037 7031 raft_consensus.cc:2454] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 0.
I20260808 04:45:04.586385 7383 leader_election.cc:302] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 389c86ef70184abe9ce69542eb732c0c, b873269318a54ee3afc18e78e628e13b; no voters:
I20260808 04:45:04.586524 7263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf0b403bb6740f0b0bbc259f3345e03" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" is_pre_election: true
I20260808 04:45:04.586628 7613 raft_consensus.cc:2781] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:04.586699 7613 raft_consensus.cc:482] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:04.586714 7263 raft_consensus.cc:2454] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 0.
I20260808 04:45:04.586758 7613 raft_consensus.cc:3037] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:04.588169 7613 raft_consensus.cc:504] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.588593 7613 leader_election.cc:288] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 election: Requested vote from peers 389c86ef70184abe9ce69542eb732c0c (127.4.217.1:38943), 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761)
I20260808 04:45:04.588940 7031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf0b403bb6740f0b0bbc259f3345e03" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389c86ef70184abe9ce69542eb732c0c"
I20260808 04:45:04.589041 7263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf0b403bb6740f0b0bbc259f3345e03" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4819b5e2593f4296a09a5ab7e5fc52df"
I20260808 04:45:04.589077 7031 raft_consensus.cc:3037] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:04.589156 7263 raft_consensus.cc:3037] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:04.590255 7031 raft_consensus.cc:2454] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 1.
I20260808 04:45:04.590255 7263 raft_consensus.cc:2454] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 1.
I20260808 04:45:04.590538 7383 leader_election.cc:302] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 389c86ef70184abe9ce69542eb732c0c, b873269318a54ee3afc18e78e628e13b; no voters:
I20260808 04:45:04.590837 7613 raft_consensus.cc:2781] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:04.590938 7613 raft_consensus.cc:686] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 1 LEADER]: Becoming Leader. State: Replica: b873269318a54ee3afc18e78e628e13b, State: Running, Role: LEADER
I20260808 04:45:04.591095 7613 consensus_queue.cc:237] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.592938 6774 catalog_manager.cc:5495] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b reported cstate change: term changed from 0 to 1, leader changed from <none> to b873269318a54ee3afc18e78e628e13b (127.4.217.3). New cstate: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } health_report { overall_health: HEALTHY } } }
Aug 08 04:45:04 dist-test-slave-2kcd krb5kdc[6694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:04.655380 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.646776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37223 (local address 127.0.0.1:58370)
0808 04:45:04.647092 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.647110 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.647241 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.647516 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.647519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.648009 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:04.648127 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.648135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.648959 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.648962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.649720 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.649731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.650179 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.650183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.650278 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.650856 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:04.650871 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:04.652364 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:04.653889 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.653897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.653902 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.654138 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.654426 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:04.654428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:04.654430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:04.654507 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:04.654872 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:04.654877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:04.654981 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.655083 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.655192 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
I20260808 04:45:04.655395 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.646856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.4.217.62:37223)
0808 04:45:04.647011 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.647015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.647194 (+ 179us) server_negotiation.cc:407] Connection header received
0808 04:45:04.647336 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.647339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.647382 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.647469 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:04.648252 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.648838 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.649845 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.650065 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.650145 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.652505 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:04.652509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:04.652511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:04.652532 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:04.653790 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.654249 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.654252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.654253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.654335 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.654589 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.654592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.654593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.654778 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:04.654864 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.655156 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.655258 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":61}
I20260808 04:45:04.659601 6774 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:58370:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:45:04.660301 6774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:04.665195 7230 tablet_service.cc:1467] Processing CreateTablet for tablet 9a83021ede6a4e8682159667fe7add8e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d199f2f475554f4aa47b4161a49615d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:45:04.665282 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 9a83021ede6a4e8682159667fe7add8e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d199f2f475554f4aa47b4161a49615d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:45:04.665282 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 9a83021ede6a4e8682159667fe7add8e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d199f2f475554f4aa47b4161a49615d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:45:04.665663 7230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a83021ede6a4e8682159667fe7add8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:04.665671 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a83021ede6a4e8682159667fe7add8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:04.665670 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a83021ede6a4e8682159667fe7add8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:04.668239 7600 tablet_bootstrap.cc:492] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: Bootstrap starting.
I20260808 04:45:04.668845 7602 tablet_bootstrap.cc:492] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: Bootstrap starting.
I20260808 04:45:04.669373 7600 tablet_bootstrap.cc:654] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:04.669727 7603 tablet_bootstrap.cc:492] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: Bootstrap starting.
I20260808 04:45:04.669883 7602 tablet_bootstrap.cc:654] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:04.670606 7603 tablet_bootstrap.cc:654] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:04.670761 7600 tablet_bootstrap.cc:492] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: No bootstrap required, opened a new log
I20260808 04:45:04.670840 7600 ts_tablet_manager.cc:1397] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:04.671408 7602 tablet_bootstrap.cc:492] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: No bootstrap required, opened a new log
I20260808 04:45:04.671362 7600 raft_consensus.cc:348] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.671495 7600 raft_consensus.cc:374] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:04.671531 7602 ts_tablet_manager.cc:1397] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 04:45:04.671548 7600 raft_consensus.cc:729] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4819b5e2593f4296a09a5ab7e5fc52df, State: Initialized, Role: FOLLOWER
I20260808 04:45:04.671686 7600 consensus_queue.cc:260] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.671808 7603 tablet_bootstrap.cc:492] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: No bootstrap required, opened a new log
I20260808 04:45:04.671885 7603 ts_tablet_manager.cc:1397] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:45:04.671988 7600 ts_tablet_manager.cc:1428] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:45:04.671959 7612 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.672045 7602 raft_consensus.cc:348] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.672134 7612 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:04.672140 7602 raft_consensus.cc:374] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:04.672174 7602 raft_consensus.cc:729] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389c86ef70184abe9ce69542eb732c0c, State: Initialized, Role: FOLLOWER
I20260808 04:45:04.672297 7602 consensus_queue.cc:260] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.672287 7603 raft_consensus.cc:348] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.672394 7603 raft_consensus.cc:374] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:04.672446 7603 raft_consensus.cc:729] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b873269318a54ee3afc18e78e628e13b, State: Initialized, Role: FOLLOWER
I20260808 04:45:04.672575 7603 consensus_queue.cc:260] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.672587 7610 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.672710 7610 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:04.672889 7603 ts_tablet_manager.cc:1428] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:04.672982 7613 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.673131 7613 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:04.674364 7602 ts_tablet_manager.cc:1428] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20260808 04:45:04.726460 7567 tablet.cc:2369] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:04.866230 7610 raft_consensus.cc:482] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:04.866407 7610 raft_consensus.cc:504] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.867602 7610 leader_election.cc:288] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761), b873269318a54ee3afc18e78e628e13b (127.4.217.3:41429)
I20260808 04:45:04.871840 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.867977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:36311 (local address 127.4.217.3:41429)
0808 04:45:04.868122 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.868127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.868387 (+ 260us) server_negotiation.cc:407] Connection header received
0808 04:45:04.868554 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.868557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.868613 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.868701 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:04.869036 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.869519 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.870668 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.871241 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.871359 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.871431 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.871627 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.871705 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:45:04.872020 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.867847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:43761 (local address 127.4.217.1:40375)
0808 04:45:04.868253 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.868266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.868408 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.868688 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.868691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.868708 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:04.868892 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.868898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.869654 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.869658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.870707 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.870718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.871632 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.871636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.871711 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.871733 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.871807 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.871868 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":46,"spinlock_wait_cycles":28416,"thread_start_us":128,"threads_started":1}
I20260808 04:45:04.872068 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.868003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:40375 (local address 127.4.217.2:43761)
0808 04:45:04.868153 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.868158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.868347 (+ 189us) server_negotiation.cc:407] Connection header received
0808 04:45:04.868515 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.868521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.868583 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.868671 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:04.869003 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.869507 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.870879 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.871472 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.871603 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.871662 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.871849 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.871936 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:45:04.872258 7613 raft_consensus.cc:482] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:04.872277 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.867877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:41429 (local address 127.4.217.1:36311)
0808 04:45:04.868294 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.868305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.868431 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.868723 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.868726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.868770 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:04.868916 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.868921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.869634 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.869639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.870539 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.870548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.871371 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.871374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.871453 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.871475 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.871603 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.871674 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":29,"spinlock_wait_cycles":3200,"thread_start_us":127,"threads_started":1}
I20260808 04:45:04.872387 7613 raft_consensus.cc:504] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.872833 7613 leader_election.cc:288] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 389c86ef70184abe9ce69542eb732c0c (127.4.217.1:38943), 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761)
I20260808 04:45:04.872838 7263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a83021ede6a4e8682159667fe7add8e" candidate_uuid: "389c86ef70184abe9ce69542eb732c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" is_pre_election: true
I20260808 04:45:04.872895 7489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a83021ede6a4e8682159667fe7add8e" candidate_uuid: "389c86ef70184abe9ce69542eb732c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b873269318a54ee3afc18e78e628e13b" is_pre_election: true
I20260808 04:45:04.872991 7263 raft_consensus.cc:2454] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 389c86ef70184abe9ce69542eb732c0c in term 0.
I20260808 04:45:04.873106 7489 raft_consensus.cc:2454] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 389c86ef70184abe9ce69542eb732c0c in term 0.
I20260808 04:45:04.873162 7031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a83021ede6a4e8682159667fe7add8e" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389c86ef70184abe9ce69542eb732c0c" is_pre_election: true
I20260808 04:45:04.873221 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a83021ede6a4e8682159667fe7add8e" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" is_pre_election: true
I20260808 04:45:04.873277 7031 raft_consensus.cc:2454] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 0.
I20260808 04:45:04.873339 7259 raft_consensus.cc:2454] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 0.
I20260808 04:45:04.873383 6914 leader_election.cc:302] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 389c86ef70184abe9ce69542eb732c0c, 4819b5e2593f4296a09a5ab7e5fc52df; no voters:
I20260808 04:45:04.873591 7383 leader_election.cc:302] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 389c86ef70184abe9ce69542eb732c0c, b873269318a54ee3afc18e78e628e13b; no voters:
I20260808 04:45:04.873705 7610 raft_consensus.cc:2781] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:04.873764 7610 raft_consensus.cc:482] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:04.873811 7610 raft_consensus.cc:3037] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:04.873883 7613 raft_consensus.cc:2781] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:04.873945 7613 raft_consensus.cc:482] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:04.873981 7613 raft_consensus.cc:3037] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:04.874922 7613 raft_consensus.cc:504] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.875236 7613 leader_election.cc:288] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 election: Requested vote from peers 389c86ef70184abe9ce69542eb732c0c (127.4.217.1:38943), 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761)
I20260808 04:45:04.875192 7610 raft_consensus.cc:504] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.875576 7610 leader_election.cc:288] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [CANDIDATE]: Term 1 election: Requested vote from peers 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761), b873269318a54ee3afc18e78e628e13b (127.4.217.3:41429)
I20260808 04:45:04.875589 7031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a83021ede6a4e8682159667fe7add8e" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389c86ef70184abe9ce69542eb732c0c"
I20260808 04:45:04.875605 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a83021ede6a4e8682159667fe7add8e" candidate_uuid: "b873269318a54ee3afc18e78e628e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4819b5e2593f4296a09a5ab7e5fc52df"
I20260808 04:45:04.875787 7031 raft_consensus.cc:2379] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b873269318a54ee3afc18e78e628e13b in current term 1: Already voted for candidate 389c86ef70184abe9ce69542eb732c0c in this term.
I20260808 04:45:04.875777 7259 raft_consensus.cc:3037] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:04.875928 7489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a83021ede6a4e8682159667fe7add8e" candidate_uuid: "389c86ef70184abe9ce69542eb732c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b873269318a54ee3afc18e78e628e13b"
I20260808 04:45:04.876014 7263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a83021ede6a4e8682159667fe7add8e" candidate_uuid: "389c86ef70184abe9ce69542eb732c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4819b5e2593f4296a09a5ab7e5fc52df"
I20260808 04:45:04.876106 7489 raft_consensus.cc:2379] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 389c86ef70184abe9ce69542eb732c0c in current term 1: Already voted for candidate b873269318a54ee3afc18e78e628e13b in this term.
I20260808 04:45:04.876575 6907 leader_election.cc:302] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 389c86ef70184abe9ce69542eb732c0c; no voters: 4819b5e2593f4296a09a5ab7e5fc52df, b873269318a54ee3afc18e78e628e13b
I20260808 04:45:04.876767 7610 raft_consensus.cc:2726] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 04:45:04.877111 7259 raft_consensus.cc:2454] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873269318a54ee3afc18e78e628e13b in term 1.
I20260808 04:45:04.877408 7383 leader_election.cc:302] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4819b5e2593f4296a09a5ab7e5fc52df, b873269318a54ee3afc18e78e628e13b; no voters: 389c86ef70184abe9ce69542eb732c0c
I20260808 04:45:04.877614 7613 raft_consensus.cc:2781] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:04.877731 7613 raft_consensus.cc:686] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [term 1 LEADER]: Becoming Leader. State: Replica: b873269318a54ee3afc18e78e628e13b, State: Running, Role: LEADER
I20260808 04:45:04.877874 7613 consensus_queue.cc:237] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } }
I20260808 04:45:04.878566 7616 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: TxnStatusTablet state changed. Reason: New leader b873269318a54ee3afc18e78e628e13b. Latest consensus state: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.878674 7616 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:45:04.878908 7671 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:45:04.879073 7671 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:45:04.879694 6771 catalog_manager.cc:5495] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b reported cstate change: term changed from 0 to 1, leader changed from <none> to b873269318a54ee3afc18e78e628e13b (127.4.217.3). New cstate: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } health_report { overall_health: HEALTHY } } }
I20260808 04:45:04.912238 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.908979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.4.217.3:41429)
0808 04:45:04.909114 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.909118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.909131 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:04.909180 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.909182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.909225 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.909292 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:04.909574 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.910063 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.911255 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.911456 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.911527 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.911830 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:04.911897 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:04.911950 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.912083 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.912128 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:45:04.912286 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.908743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:41429 (local address 127.0.0.1:33138)
0808 04:45:04.908959 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.908976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.909084 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.909343 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.909347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.909357 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:04.909464 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.909471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.910169 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.910173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.911133 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.911144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.911555 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.911559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.911649 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.911701 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:04.911980 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:04.911994 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.912063 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.912143 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 04:45:04.915074 7259 raft_consensus.cc:1261] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Refusing update from remote peer b873269318a54ee3afc18e78e628e13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:04.915176 7031 raft_consensus.cc:1261] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Refusing update from remote peer b873269318a54ee3afc18e78e628e13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:04.915601 7616 consensus_queue.cc:1035] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:04.915750 7613 consensus_queue.cc:1035] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:04.917168 7610 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: TxnStatusTablet state changed. Reason: New leader b873269318a54ee3afc18e78e628e13b. Latest consensus state: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.917279 7610 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:04.917433 7612 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: TxnStatusTablet state changed. Reason: New leader b873269318a54ee3afc18e78e628e13b. Latest consensus state: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.917537 7612 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:04.919478 7613 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.919600 7613 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:45:04.920753 7610 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.920882 7610 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P 389c86ef70184abe9ce69542eb732c0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:04.922788 7616 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.922935 7616 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P b873269318a54ee3afc18e78e628e13b: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:45:04.923746 7624 mvcc.cc:205] Tried to move back new op lower bound from 7316128992927715328 to 7316128992781234176. Current Snapshot: MvccSnapshot[applied={T|T < 7316128992927715328}]
I20260808 04:45:04.924227 7612 tablet_replica.cc:439] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b873269318a54ee3afc18e78e628e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 } } peers { permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 } } peers { permanent_uuid: "b873269318a54ee3afc18e78e628e13b" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 41429 } } }
I20260808 04:45:04.924326 7612 tablet_replica.cc:442] T 9a83021ede6a4e8682159667fe7add8e P 4819b5e2593f4296a09a5ab7e5fc52df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:04.934403 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.930909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:40111 (local address 127.4.217.3:41429)
0808 04:45:04.931026 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.931030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.931190 (+ 160us) server_negotiation.cc:407] Connection header received
0808 04:45:04.931274 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.931276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.931328 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.931413 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:04.931624 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.932405 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.933272 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.933609 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.933711 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.933974 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:04.934048 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:04.934119 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.934200 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.934247 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":44}
I20260808 04:45:04.934384 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.930845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:41429 (local address 127.4.217.3:40111)
0808 04:45:04.931106 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:04.931119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:04.931207 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:04.931434 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:04.931436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:04.931443 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:04.931525 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.931529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.932508 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.932510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.933163 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:04.933172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.933722 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.933724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:04.933810 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.933852 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:04.934137 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:04.934142 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:45:04.934190 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 04:45:04.934262 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":19,"thread_start_us":81,"threads_started":1}
I20260808 04:45:04.941200 7259 raft_consensus.cc:1261] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Refusing update from remote peer b873269318a54ee3afc18e78e628e13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:04.941200 7031 raft_consensus.cc:1261] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Refusing update from remote peer b873269318a54ee3afc18e78e628e13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:04.941576 7682 consensus_queue.cc:1035] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "389c86ef70184abe9ce69542eb732c0c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:04.941687 7616 consensus_queue.cc:1035] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4819b5e2593f4296a09a5ab7e5fc52df" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:04.945025 7623 mvcc.cc:205] Tried to move back new op lower bound from 7316128993035472896 to 7316128991606542336. Current Snapshot: MvccSnapshot[applied={T|T < 7316128993035472896}]
I20260808 04:45:04.979676 6771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:04.979825 6771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:04.981864 6771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25461ec4d20b4e3284f8b758a6140f04: Sending DeleteTablet for 3 replicas of tablet 1bf0b403bb6740f0b0bbc259f3345e03
I20260808 04:45:04.982460 7230 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf0b403bb6740f0b0bbc259f3345e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:04 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:33958
I20260808 04:45:04.982506 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf0b403bb6740f0b0bbc259f3345e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:04 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:33110
I20260808 04:45:04.982681 7636 tablet_replica.cc:330] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: stopping tablet replica
I20260808 04:45:04.982676 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf0b403bb6740f0b0bbc259f3345e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:04 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:42186
I20260808 04:45:04.982825 7636 raft_consensus.cc:2229] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:04.982872 7638 tablet_replica.cc:330] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: stopping tablet replica
I20260808 04:45:04.982883 7635 tablet_replica.cc:330] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: stopping tablet replica
I20260808 04:45:04.982982 7638 raft_consensus.cc:2229] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:04.982982 7635 raft_consensus.cc:2229] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:04.983013 7636 raft_consensus.cc:2258] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:04.983098 7635 raft_consensus.cc:2258] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:04.983098 7638 raft_consensus.cc:2258] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:04.983909 7638 ts_tablet_manager.cc:1905] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:04.983935 7636 ts_tablet_manager.cc:1905] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:04.986295 7636 ts_tablet_manager.cc:1918] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:04.986351 7636 log.cc:1196] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/1bf0b403bb6740f0b0bbc259f3345e03
I20260808 04:45:04.986538 7636 ts_tablet_manager.cc:1939] T 1bf0b403bb6740f0b0bbc259f3345e03 P b873269318a54ee3afc18e78e628e13b: Deleting consensus metadata
I20260808 04:45:04.986752 7638 ts_tablet_manager.cc:1918] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:04.986826 7638 log.cc:1196] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/1bf0b403bb6740f0b0bbc259f3345e03
I20260808 04:45:04.987087 7638 ts_tablet_manager.cc:1939] T 1bf0b403bb6740f0b0bbc259f3345e03 P 389c86ef70184abe9ce69542eb732c0c: Deleting consensus metadata
I20260808 04:45:04.987139 6747 catalog_manager.cc:4841] TS b873269318a54ee3afc18e78e628e13b (127.4.217.3:41429): tablet 1bf0b403bb6740f0b0bbc259f3345e03 (table test-table [id=f12399a3c37b48f799eca1634635ca91]) successfully deleted
I20260808 04:45:04.987510 7635 ts_tablet_manager.cc:1905] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:04.987741 6753 catalog_manager.cc:4841] TS 389c86ef70184abe9ce69542eb732c0c (127.4.217.1:38943): tablet 1bf0b403bb6740f0b0bbc259f3345e03 (table test-table [id=f12399a3c37b48f799eca1634635ca91]) successfully deleted
I20260808 04:45:04.990139 7635 ts_tablet_manager.cc:1918] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:04.990242 7635 log.cc:1196] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/1bf0b403bb6740f0b0bbc259f3345e03
I20260808 04:45:04.990502 7635 ts_tablet_manager.cc:1939] T 1bf0b403bb6740f0b0bbc259f3345e03 P 4819b5e2593f4296a09a5ab7e5fc52df: Deleting consensus metadata
I20260808 04:45:04.991202 6753 catalog_manager.cc:4841] TS 4819b5e2593f4296a09a5ab7e5fc52df (127.4.217.2:43761): tablet 1bf0b403bb6740f0b0bbc259f3345e03 (table test-table [id=f12399a3c37b48f799eca1634635ca91]) successfully deleted
Aug 08 04:45:04 dist-test-slave-2kcd krb5kdc[6694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:45:04.998708 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:04.988270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.4.217.1:38943)
0808 04:45:04.988494 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:04.988498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:04.989094 (+ 596us) server_negotiation.cc:407] Connection header received
0808 04:45:04.989223 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:04.989227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:04.989280 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:04.989382 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:04.990251 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.991060 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.991893 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:04.992133 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:04.992250 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:04.994955 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:04.994966 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:04.994980 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:04.995003 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:04.996654 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.997232 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.997238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.997240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.997331 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:04.997683 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:04.997687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:04.997689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:04.997952 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:04.998200 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:04.998378 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 04:45:04.998507 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
W20260808 04:45:04.999197 7095 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42202
I20260808 04:45:05.010651 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:05.001203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.4.217.62:37223)
0808 04:45:05.001344 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:05.001349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:05.001547 (+ 198us) server_negotiation.cc:407] Connection header received
0808 04:45:05.001638 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:05.001640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:05.001685 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:05.001749 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:05.002658 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:05.003377 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:05.004544 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:05.004916 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:05.005018 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:05.006642 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:05.006646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:05.006649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:05.006678 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:05.008554 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:05.009232 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:05.009235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:05.009237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:05.009374 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:05.009714 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:05.009717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:05.009719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:05.009933 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:05.010063 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:05.010354 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 04:45:05.010487 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 04:45:05.011230 6771 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58384
I20260808 04:45:05.012702 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6883
I20260808 04:45:05.048530 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 7122
I20260808 04:45:05.056193 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 7348
I20260808 04:45:05.072304 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6718
2026-08-08T04:45:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3181 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:05 dist-test-slave-2kcd krb5kdc[7699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:05 dist-test-slave-2kcd krb5kdc[7699](info): set up 2 sockets
Aug 08 04:45:05 dist-test-slave-2kcd krb5kdc[7699](info): commencing operation
krb5kdc: starting...
W20260808 04:45:06.346196 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:06 dist-test-slave-2kcd krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:06.473527 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35989
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:06.585939 7724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:06.586198 7724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:06.586236 7724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:06.590082 7724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:06.590123 7724 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:45:06.590139 7724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:06.590155 7724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:06.590171 7724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:06.590185 7724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:06.595189 7724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35989
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:06.596117 7724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:06.597098 7724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:45:06.605698 7724 server_base.cc:1034] running on GCE node
W20260808 04:45:06.605648 7734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:06.605648 7738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:06.605654 7736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:06.606582 7724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:06.607673 7724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:06.608861 7724 hybrid_clock.cc:648] HybridClock initialized: now 1786164306608833 us; error 29 us; skew 500 ppm
Aug 08 04:45:06 dist-test-slave-2kcd krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:06.615681 7724 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:06.616581 7724 webserver.cc:460] Webserver started at http://127.4.217.62:44777/ using document root <none> and password file <none>
I20260808 04:45:06.617230 7724 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:06.617327 7724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:06.617578 7724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:06.619971 7724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "75298fafbc304a92ba126b8cec14a3e8"
format_stamp: "Formatted at 2026-08-08 04:45:06 on dist-test-slave-2kcd"
server_key: "91951b2e4650c9293824042de62138df"
server_key_iv: "68bb8090c5f25f7d2018d0238194ae5f"
server_key_version: "encryptionkey@0"
I20260808 04:45:06.620679 7724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "75298fafbc304a92ba126b8cec14a3e8"
format_stamp: "Formatted at 2026-08-08 04:45:06 on dist-test-slave-2kcd"
server_key: "91951b2e4650c9293824042de62138df"
server_key_iv: "68bb8090c5f25f7d2018d0238194ae5f"
server_key_version: "encryptionkey@0"
I20260808 04:45:06.625125 7724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:45:06.627492 7748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:06.628695 7724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:06.628860 7724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "75298fafbc304a92ba126b8cec14a3e8"
format_stamp: "Formatted at 2026-08-08 04:45:06 on dist-test-slave-2kcd"
server_key: "91951b2e4650c9293824042de62138df"
server_key_iv: "68bb8090c5f25f7d2018d0238194ae5f"
server_key_version: "encryptionkey@0"
I20260808 04:45:06.628980 7724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:06.644235 7724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:06.647004 7724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:06.647217 7724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:06.655067 7724 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:35989
I20260808 04:45:06.655097 7850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:35989 every 8 connection(s)
I20260808 04:45:06.655997 7724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:06.658776 7856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:06.659703 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 7724
I20260808 04:45:06.659816 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:06.660135 4964 external_mini_cluster.cc:1428] Setting key bbbf31046c7ae303120e2e07cc0b12f5
I20260808 04:45:06.665258 7856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8: Bootstrap starting.
I20260808 04:45:06.667459 7856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:06.668171 7856 log.cc:824] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:45:06 dist-test-slave-2kcd krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:06.670281 7856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8: No bootstrap required, opened a new log
I20260808 04:45:06.674477 7856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75298fafbc304a92ba126b8cec14a3e8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35989 } }
I20260808 04:45:06.674752 7856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:06.674820 7856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75298fafbc304a92ba126b8cec14a3e8, State: Initialized, Role: FOLLOWER
I20260808 04:45:06.675290 7856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75298fafbc304a92ba126b8cec14a3e8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35989 } }
I20260808 04:45:06.675439 7856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:06.675510 7856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:06.675619 7856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:06.675803 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:06.661573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.4.217.62:35989)
0808 04:45:06.662041 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:06.662056 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:06.662105 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:45:06.662813 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:06.662832 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:06.663352 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:06.663745 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:06.664568 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.665708 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.666581 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.667053 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.667233 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:06.669883 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:06.669890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:06.669905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:06.669938 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:06.672742 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:06.673200 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:06.673206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:06.673231 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:06.673373 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:06.673642 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:06.673645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:06.673647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:06.674157 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:06.674368 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:06.674561 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 04:45:06.674858 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":148,"threads_started":1}
I20260808 04:45:06.676671 7856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75298fafbc304a92ba126b8cec14a3e8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35989 } }
I20260808 04:45:06.677024 7856 leader_election.cc:302] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75298fafbc304a92ba126b8cec14a3e8; no voters:
I20260808 04:45:06.677356 7856 leader_election.cc:288] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:06.677498 7862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:06.677731 7862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [term 1 LEADER]: Becoming Leader. State: Replica: 75298fafbc304a92ba126b8cec14a3e8, State: Running, Role: LEADER
I20260808 04:45:06.678023 7862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75298fafbc304a92ba126b8cec14a3e8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35989 } }
I20260808 04:45:06.678735 7856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:06.679114 7862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75298fafbc304a92ba126b8cec14a3e8. Latest consensus state: current_term: 1 leader_uuid: "75298fafbc304a92ba126b8cec14a3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75298fafbc304a92ba126b8cec14a3e8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35989 } } }
I20260808 04:45:06.679239 7862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:06.679639 7866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:06.679653 7863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75298fafbc304a92ba126b8cec14a3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75298fafbc304a92ba126b8cec14a3e8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35989 } } }
I20260808 04:45:06.679805 7863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:06.682677 7866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:06.688012 7866 catalog_manager.cc:1347] Generated new cluster ID: 0b68c4d0630f471aa1251f6c0f6da10a
I20260808 04:45:06.688086 7866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:06.698179 7866 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:06.699136 7866 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:06.703428 7866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8: Generated new TSK 0
I20260808 04:45:06.703984 7866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:45:06.737136 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35989
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:06.853178 7892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:06.853422 7892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:06.853509 7892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:06.857566 7892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:06.857621 7892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:06.857690 7892 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:45:06.857729 7892 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:45:06.857795 7892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:06.863092 7892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.217.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:06.864126 7892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:06.865211 7892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:06.875313 7908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:06.875325 7897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:06.875497 7892 server_base.cc:1034] running on GCE node
W20260808 04:45:06.875366 7901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:06.875967 7892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:06.876605 7892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:06.877796 7892 hybrid_clock.cc:648] HybridClock initialized: now 1786164306877796 us; error 48 us; skew 500 ppm
Aug 08 04:45:06 dist-test-slave-2kcd krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:06.880663 7892 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:45:06.881562 7892 webserver.cc:460] Webserver started at http://127.4.217.1:36707/ using document root <none> and password file <none>
I20260808 04:45:06.882121 7892 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:06.882213 7892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:06.882448 7892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:06.884465 7892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "c805e12ed3304184bb93c3b5453d06ec"
format_stamp: "Formatted at 2026-08-08 04:45:06 on dist-test-slave-2kcd"
server_key: "9f57366cc8c3b79996530723820de110"
server_key_iv: "3f47b1c01d0159d7d6f0681be9c69df7"
server_key_version: "encryptionkey@0"
I20260808 04:45:06.885043 7892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "c805e12ed3304184bb93c3b5453d06ec"
format_stamp: "Formatted at 2026-08-08 04:45:06 on dist-test-slave-2kcd"
server_key: "9f57366cc8c3b79996530723820de110"
server_key_iv: "3f47b1c01d0159d7d6f0681be9c69df7"
server_key_version: "encryptionkey@0"
I20260808 04:45:06.888764 7892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:45:06.891147 7918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:06.892323 7892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:45:06.892469 7892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "c805e12ed3304184bb93c3b5453d06ec"
format_stamp: "Formatted at 2026-08-08 04:45:06 on dist-test-slave-2kcd"
server_key: "9f57366cc8c3b79996530723820de110"
server_key_iv: "3f47b1c01d0159d7d6f0681be9c69df7"
server_key_version: "encryptionkey@0"
I20260808 04:45:06.892578 7892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:06.903118 7892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:06.906009 7892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:06.906250 7892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:06.907709 7892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:06.907791 7892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:06.907861 7892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:06.907913 7892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:45:06 dist-test-slave-2kcd krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:06.919421 7892 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:34939
I20260808 04:45:06.919495 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:34939 every 8 connection(s)
I20260808 04:45:06.920506 7892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:06.923149 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:06.908526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:36017 (local address 127.4.217.62:35989)
0808 04:45:06.908665 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:06.908668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:06.909696 (+ 1028us) server_negotiation.cc:407] Connection header received
0808 04:45:06.910774 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:06.910781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:06.910849 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:06.910965 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:06.912740 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.913687 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.915514 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.915922 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.916052 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:06.919264 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:06.919267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:06.919269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:06.919293 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:06.920910 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:06.921516 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:06.921519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:06.921521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:06.921599 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:06.921978 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:06.921981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:06.921982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:06.922187 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:06.922296 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:06.922910 (+ 614us) server_negotiation.cc:299] Negotiation successful
0808 04:45:06.923028 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:45:06.923907 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 7892
I20260808 04:45:06.924034 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:06.924074 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:06.909084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35989 (local address 127.4.217.1:36017)
0808 04:45:06.909554 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:06.909595 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:06.910517 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:06.911304 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:06.911316 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:06.911843 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:06.912552 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:06.912578 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.913830 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.913835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:06.915369 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:06.915384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.916225 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.916228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:06.916367 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:06.916983 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:06.917011 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:06.919012 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:06.921075 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:06.921080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:06.921093 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:06.921438 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:06.921667 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:06.921669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:06.921670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:06.921828 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:06.922308 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:06.922314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:06.922589 (+ 275us) client_negotiation.cc:769] Sending connection context
0808 04:45:06.922784 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 04:45:06.923103 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
I20260808 04:45:06.924348 4964 external_mini_cluster.cc:1428] Setting key b57d1c46e2e99db3bc792d09a827cb3a
I20260808 04:45:06.925776 8110 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35989
I20260808 04:45:06.926011 8110 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:06.926591 8110 heartbeater.cc:511] Master 127.4.217.62:35989 requested a full tablet report, sending...
I20260808 04:45:06.928205 7774 ts_manager.cc:194] Registered new tserver with Master: c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939)
I20260808 04:45:06.929615 7774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:36017
I20260808 04:45:06.936437 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:06.928328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35989 (local address 127.4.217.1:47455)
0808 04:45:06.928662 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:06.928680 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:06.928812 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:06.929082 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:06.929087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:06.929325 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:06.929485 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:06.929492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.930438 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.930440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:06.931012 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:06.931020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.931654 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.931657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:06.931726 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:06.932308 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:06.932323 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:06.932829 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:06.934977 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:06.934980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:06.934981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:06.935180 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:06.935486 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:06.935488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:06.935490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:06.935565 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:06.936023 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:06.936025 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:06.936108 (+ 83us) client_negotiation.cc:769] Sending connection context
0808 04:45:06.936185 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:45:06.936287 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":120,"threads_started":1}
I20260808 04:45:06.936604 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:06.928125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:47455 (local address 127.4.217.62:35989)
0808 04:45:06.928235 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:06.928240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:06.928767 (+ 527us) server_negotiation.cc:407] Connection header received
0808 04:45:06.928901 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:06.928905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:06.928951 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:06.929037 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:06.929616 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.930318 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.931144 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:06.931535 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:06.931660 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:06.933004 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:06.933010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:06.933017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:06.933050 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:06.934850 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:06.935289 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:06.935293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:06.935295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:06.935389 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:06.935660 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:06.935666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:06.935668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:06.935915 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:06.936021 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:06.936277 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 04:45:06.936409 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:45:06.956116 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35989
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:07.072638 8125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:07.072918 8125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:07.073009 8125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:07.077080 8125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:07.077142 8125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:07.077226 8125 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:45:07.077272 8125 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:45:07.077354 8125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:07.082911 8125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.217.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:07.083950 8125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:07.085034 8125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:07.095188 8137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:07.095185 8139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:07.095180 8136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:07.095316 8125 server_base.cc:1034] running on GCE node
I20260808 04:45:07.095811 8125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:07.096509 8125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:07.097674 8125 hybrid_clock.cc:648] HybridClock initialized: now 1786164307097640 us; error 51 us; skew 500 ppm
Aug 08 04:45:07 dist-test-slave-2kcd krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164307, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:07.100356 8125 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:45:07.101249 8125 webserver.cc:460] Webserver started at http://127.4.217.2:33503/ using document root <none> and password file <none>
I20260808 04:45:07.101775 8125 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:07.101835 8125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:07.102002 8125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:07.104050 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "f8619b1fc8cf4fd4b196b01da15a5b43"
format_stamp: "Formatted at 2026-08-08 04:45:07 on dist-test-slave-2kcd"
server_key: "b4f6e6238dc447536dbbf9102dd657b8"
server_key_iv: "9c1bb9e00679158b8e2b968cfb6cad71"
server_key_version: "encryptionkey@0"
I20260808 04:45:07.104552 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "f8619b1fc8cf4fd4b196b01da15a5b43"
format_stamp: "Formatted at 2026-08-08 04:45:07 on dist-test-slave-2kcd"
server_key: "b4f6e6238dc447536dbbf9102dd657b8"
server_key_iv: "9c1bb9e00679158b8e2b968cfb6cad71"
server_key_version: "encryptionkey@0"
I20260808 04:45:07.108281 8125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:45:07.110690 8153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:07.111876 8125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:07.111971 8125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "f8619b1fc8cf4fd4b196b01da15a5b43"
format_stamp: "Formatted at 2026-08-08 04:45:07 on dist-test-slave-2kcd"
server_key: "b4f6e6238dc447536dbbf9102dd657b8"
server_key_iv: "9c1bb9e00679158b8e2b968cfb6cad71"
server_key_version: "encryptionkey@0"
I20260808 04:45:07.112063 8125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:07.136588 8125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:07.139451 8125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:07.139611 8125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:07.141145 8125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:07.141196 8125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:07.141263 8125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:07.141296 8125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:45:07 dist-test-slave-2kcd krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164307, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:07.153313 8125 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:44853
I20260808 04:45:07.153398 8337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:44853 every 8 connection(s)
I20260808 04:45:07.154232 8125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:07.155961 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.141990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:52787 (local address 127.4.217.62:35989)
0808 04:45:07.142152 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.142156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.143077 (+ 921us) server_negotiation.cc:407] Connection header received
0808 04:45:07.144237 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.144243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.144314 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.144382 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:07.145972 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.146640 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.147927 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.148161 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.148281 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.152084 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:07.152089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:07.152091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:07.152126 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:07.153792 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.154313 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.154318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.154320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.154408 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.154791 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.154795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.154797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.155029 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:07.155162 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.155729 (+ 567us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.155837 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:45:07.156764 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.142405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35989 (local address 127.4.217.2:52787)
0808 04:45:07.142905 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.142955 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.143972 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.144896 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.144906 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.145238 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:07.145795 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.145834 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.146782 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.146786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.147785 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.147796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.148335 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.148339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.148509 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.149193 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:07.149212 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:07.151902 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:07.153915 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.153920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.153934 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.154198 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.154516 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.154519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.154521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.154686 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.155164 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:07.155169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:07.155462 (+ 293us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.155673 (+ 211us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.155914 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":173,"threads_started":1}
I20260808 04:45:07.158077 8340 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35989
I20260808 04:45:07.158244 8340 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:07.158785 8340 heartbeater.cc:511] Master 127.4.217.62:35989 requested a full tablet report, sending...
I20260808 04:45:07.159827 7776 ts_manager.cc:194] Registered new tserver with Master: f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2:44853)
I20260808 04:45:07.160544 7776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:52787
I20260808 04:45:07.163273 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 8125
I20260808 04:45:07.163396 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:07.163714 4964 external_mini_cluster.cc:1428] Setting key 9edccc09a7ee6d794791d33a07fc7d92
I20260808 04:45:07.167929 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.160203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:56391 (local address 127.4.217.62:35989)
0808 04:45:07.160320 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.160323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.161036 (+ 713us) server_negotiation.cc:407] Connection header received
0808 04:45:07.161128 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.161133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.161179 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.161250 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:07.161748 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.162219 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.163087 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.163415 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.163523 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.164688 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:07.164692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:07.164694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:07.164717 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:07.166146 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.166654 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.166656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.166657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.166728 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.167031 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.167033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.167034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.167282 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:07.167396 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.167669 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.167775 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:45:07.167937 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.160115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35989 (local address 127.4.217.2:56391)
0808 04:45:07.160953 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.160965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.161043 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.161305 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.161307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.161518 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:07.161634 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.161641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.162351 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.162354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.162964 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.162974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.163550 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.163553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.163622 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.164151 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:07.164162 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:07.164558 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:07.166263 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.166269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.166274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.166553 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.166818 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.166824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.166829 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.166934 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.167388 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:07.167391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:07.167492 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.167612 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.167752 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:45:07.192313 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35989
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:07.307775 8355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:07.308020 8355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:07.308100 8355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:07.312135 8355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:07.312192 8355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:07.312264 8355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:45:07.312296 8355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:45:07.312381 8355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:07.317773 8355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.217.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:07.318768 8355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:07.319800 8355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:07.329689 8363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:07.329802 8366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:07.329689 8364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:07.330071 8355 server_base.cc:1034] running on GCE node
I20260808 04:45:07.330478 8355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:07.331130 8355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:07.332316 8355 hybrid_clock.cc:648] HybridClock initialized: now 1786164307332288 us; error 47 us; skew 500 ppm
Aug 08 04:45:07 dist-test-slave-2kcd krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164307, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:07.335124 8355 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:45:07.335881 8355 webserver.cc:460] Webserver started at http://127.4.217.3:43975/ using document root <none> and password file <none>
I20260808 04:45:07.336398 8355 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:07.336454 8355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:07.336616 8355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:07.338702 8355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "d033aeb702704f30b07e382d474d3271"
format_stamp: "Formatted at 2026-08-08 04:45:07 on dist-test-slave-2kcd"
server_key: "55c207812dfbdd58529100543a7cecb1"
server_key_iv: "79cf23dd75f8af6c515a13e1dd2e6f73"
server_key_version: "encryptionkey@0"
I20260808 04:45:07.339327 8355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "d033aeb702704f30b07e382d474d3271"
format_stamp: "Formatted at 2026-08-08 04:45:07 on dist-test-slave-2kcd"
server_key: "55c207812dfbdd58529100543a7cecb1"
server_key_iv: "79cf23dd75f8af6c515a13e1dd2e6f73"
server_key_version: "encryptionkey@0"
I20260808 04:45:07.342893 8355 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:45:07.345211 8375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:07.346393 8355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:07.346540 8355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "d033aeb702704f30b07e382d474d3271"
format_stamp: "Formatted at 2026-08-08 04:45:07 on dist-test-slave-2kcd"
server_key: "55c207812dfbdd58529100543a7cecb1"
server_key_iv: "79cf23dd75f8af6c515a13e1dd2e6f73"
server_key_version: "encryptionkey@0"
I20260808 04:45:07.346652 8355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:07.356734 8355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:07.359756 8355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:07.359961 8355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:07.361486 8355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:07.361562 8355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:07.361637 8355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:07.361680 8355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:45:07 dist-test-slave-2kcd krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164307, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:07.373315 8355 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:43355
I20260808 04:45:07.373406 8556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:43355 every 8 connection(s)
I20260808 04:45:07.374222 8355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:07.377453 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.362288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:58715 (local address 127.4.217.62:35989)
0808 04:45:07.362462 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.362468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.364911 (+ 2443us) server_negotiation.cc:407] Connection header received
0808 04:45:07.366045 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.366049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.366114 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.366202 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:07.367915 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.368857 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.370721 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.370972 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.371079 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.373922 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:07.373927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:07.373929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:07.373958 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:07.375621 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.376077 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.376080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.376081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.376158 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.376464 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.376466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.376467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.376643 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:07.376721 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.377235 (+ 514us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.377331 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:45:07.378213 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.363061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35989 (local address 127.4.217.3:58715)
0808 04:45:07.364728 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.364813 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.365759 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.366422 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.366433 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.366962 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:07.367700 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.367720 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.368999 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.369003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.370570 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.370582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.371119 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.371123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.371303 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.371922 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:07.371941 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:07.373750 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:07.375723 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.375728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.375738 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.375989 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.376239 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.376241 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.376243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.376368 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.376733 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:07.376738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:07.377022 (+ 284us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.377183 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.377414 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1466,"thread_start_us":182,"threads_started":1}
I20260808 04:45:07.378552 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 8355
I20260808 04:45:07.378696 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:07.379047 4964 external_mini_cluster.cc:1428] Setting key 7fe82dab07d1f77278bb2a7e1056c69b
I20260808 04:45:07.379680 8557 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35989
I20260808 04:45:07.379825 8557 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:07.380417 8557 heartbeater.cc:511] Master 127.4.217.62:35989 requested a full tablet report, sending...
I20260808 04:45:07.381517 7774 ts_manager.cc:194] Registered new tserver with Master: d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:07.382262 7774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:58715
I20260808 04:45:07.389004 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.382019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:51781 (local address 127.4.217.62:35989)
0808 04:45:07.382154 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.382158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.382330 (+ 172us) server_negotiation.cc:407] Connection header received
0808 04:45:07.382488 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.382490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.382570 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.382650 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:07.383159 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.383911 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.384702 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.384979 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.385067 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.386021 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:07.386024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:07.386026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:07.386050 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:07.387590 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.387947 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.387949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.387950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.388024 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.388259 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.388261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.388262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.388452 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:07.388540 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.388745 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.388888 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:45:07.389027 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.381953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35989 (local address 127.4.217.3:51781)
0808 04:45:07.382241 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.382256 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.382361 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.382686 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.382689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.382905 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:07.383041 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.383047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.384039 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.384041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.384594 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.384601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.385086 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.385088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.385159 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.385583 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:07.385594 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:07.385895 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:07.387685 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.387687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.387689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.387863 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.388105 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.388107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.388108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.388181 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.388542 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:07.388544 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:07.388627 (+ 83us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.388709 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.388876 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":4,"thread_start_us":124,"threads_started":1}
I20260808 04:45:07.393011 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:45:07 dist-test-slave-2kcd krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:45:07 dist-test-slave-2kcd krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:07.421135 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.412624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.4.217.62:35989)
0808 04:45:07.412774 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.412839 (+ 65us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.412855 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:45:07.412906 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.412909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.412962 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.413058 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:07.413780 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.414325 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.415293 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.415521 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.415630 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.418056 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:07.418060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:07.418063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:07.418094 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:07.419411 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.419854 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.419858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.419860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.419942 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.420229 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.420233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.420235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.420417 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:07.420567 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.420835 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.420959 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:45:07.424037 7774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:07.427258 7774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:07.442135 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.435976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:34939 (local address 127.0.0.1:51294)
0808 04:45:07.436428 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.436483 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.436632 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.437514 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.437520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.437559 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:07.437708 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.437715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.439135 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.439139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.440458 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.440466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.441561 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.441565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.441681 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.441727 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.441842 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.441964 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":90,"threads_started":1}
I20260808 04:45:07.442138 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.436126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.4.217.1:34939)
0808 04:45:07.436926 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.436935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.436961 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:45:07.437053 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.437058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.437278 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.437435 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:07.437844 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.438995 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.440601 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.441420 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.441530 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.441694 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.441892 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.441997 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":108,"threads_started":1}
I20260808 04:45:07.442456 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.436029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:44853 (local address 127.0.0.1:42970)
0808 04:45:07.437596 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.437610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.437775 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.438263 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.438266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.438284 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:07.438396 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.438402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.439717 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.439722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.440939 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.440951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.442055 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.442059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.442145 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.442176 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.442254 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.442322 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":89,"threads_started":1}
I20260808 04:45:07.442880 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.436900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.4.217.2:44853)
0808 04:45:07.437220 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.437232 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.437702 (+ 470us) server_negotiation.cc:407] Connection header received
0808 04:45:07.437923 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.437928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.438110 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.438260 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:07.438527 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.439557 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.441054 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.441701 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.441817 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.441981 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.442692 (+ 711us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.442770 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":119,"threads_started":1}
I20260808 04:45:07.443785 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.437041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.4.217.3:43355)
0808 04:45:07.437989 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.437997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.438014 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:45:07.438076 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.438079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.438241 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.438386 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:07.438756 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.440250 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.441728 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.442926 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.443053 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.443203 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.443530 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.443637 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":120,"threads_started":1}
I20260808 04:45:07.445219 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.436798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:43355 (local address 127.0.0.1:55110)
0808 04:45:07.437414 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.437427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.437624 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.438551 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.438553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.438568 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:07.438642 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.438646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.440379 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.440382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.441589 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.441600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.443269 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.443272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.443368 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.443391 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.443474 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.445094 (+ 1620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":97,"threads_started":1}
I20260808 04:45:07.445523 8000 tablet_service.cc:1467] Processing CreateTablet for tablet 7054cb2f12284db5b00db1e38feb8d11 (DEFAULT_TABLE table=test-table [id=7ff409d9618544c9a45a57af62f4f88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:07.445535 8237 tablet_service.cc:1467] Processing CreateTablet for tablet 7054cb2f12284db5b00db1e38feb8d11 (DEFAULT_TABLE table=test-table [id=7ff409d9618544c9a45a57af62f4f88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:07.446727 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7054cb2f12284db5b00db1e38feb8d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:07.446727 8000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7054cb2f12284db5b00db1e38feb8d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:07.451320 8464 tablet_service.cc:1467] Processing CreateTablet for tablet 7054cb2f12284db5b00db1e38feb8d11 (DEFAULT_TABLE table=test-table [id=7ff409d9618544c9a45a57af62f4f88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:07.452443 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7054cb2f12284db5b00db1e38feb8d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:07.454784 8598 tablet_bootstrap.cc:492] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: Bootstrap starting.
I20260808 04:45:07.455966 8597 tablet_bootstrap.cc:492] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: Bootstrap starting.
I20260808 04:45:07.457489 8598 tablet_bootstrap.cc:654] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:07.458060 8597 tablet_bootstrap.cc:654] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:07.458408 8598 log.cc:824] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:07.458890 8597 log.cc:824] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:07.460013 8602 tablet_bootstrap.cc:492] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: Bootstrap starting.
I20260808 04:45:07.460599 8598 tablet_bootstrap.cc:492] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: No bootstrap required, opened a new log
I20260808 04:45:07.460826 8598 ts_tablet_manager.cc:1397] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:45:07.460871 8597 tablet_bootstrap.cc:492] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: No bootstrap required, opened a new log
I20260808 04:45:07.461042 8597 ts_tablet_manager.cc:1397] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:45:07.462206 8602 tablet_bootstrap.cc:654] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:07.463035 8602 log.cc:824] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:07.465494 8597 raft_consensus.cc:348] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.465494 8598 raft_consensus.cc:348] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.465772 8597 raft_consensus.cc:374] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:07.465772 8598 raft_consensus.cc:374] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:07.465837 8597 raft_consensus.cc:729] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c805e12ed3304184bb93c3b5453d06ec, State: Initialized, Role: FOLLOWER
I20260808 04:45:07.465847 8598 raft_consensus.cc:729] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8619b1fc8cf4fd4b196b01da15a5b43, State: Initialized, Role: FOLLOWER
I20260808 04:45:07.466325 8597 consensus_queue.cc:260] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.466325 8598 consensus_queue.cc:260] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.467342 8598 ts_tablet_manager.cc:1428] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:45:07.467342 8597 ts_tablet_manager.cc:1428] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:45:07.467391 8602 tablet_bootstrap.cc:492] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: No bootstrap required, opened a new log
I20260808 04:45:07.467607 8602 ts_tablet_manager.cc:1397] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260808 04:45:07.467723 8110 heartbeater.cc:503] Master 127.4.217.62:35989 was elected leader, sending a full tablet report...
I20260808 04:45:07.468708 8340 heartbeater.cc:503] Master 127.4.217.62:35989 was elected leader, sending a full tablet report...
I20260808 04:45:07.471498 8602 raft_consensus.cc:348] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.471774 8602 raft_consensus.cc:374] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:07.471839 8602 raft_consensus.cc:729] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d033aeb702704f30b07e382d474d3271, State: Initialized, Role: FOLLOWER
I20260808 04:45:07.472306 8602 consensus_queue.cc:260] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.473100 8557 heartbeater.cc:503] Master 127.4.217.62:35989 was elected leader, sending a full tablet report...
I20260808 04:45:07.473274 8602 ts_tablet_manager.cc:1428] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260808 04:45:07.625308 8559 tablet.cc:2369] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:45:07.655433 8342 tablet.cc:2369] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:45:07.672071 8111 tablet.cc:2369] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:07.744604 8605 raft_consensus.cc:482] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:07.744880 8605 raft_consensus.cc:504] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.746102 8605 leader_election.cc:288] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2:44853), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:07.750379 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.746427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:52821 (local address 127.4.217.2:44853)
0808 04:45:07.746703 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.746709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.746750 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:45:07.746916 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.746920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.746978 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.747079 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:07.747399 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.747945 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.749217 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.749620 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.749765 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.749818 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.750168 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.750249 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260808 04:45:07.750386 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.746313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:44853 (local address 127.4.217.1:52821)
0808 04:45:07.746659 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.746675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.746787 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.747096 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.747099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.747118 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:07.747272 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.747279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.748065 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.748072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.749094 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.749105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.749917 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.749921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.750013 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.750051 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.750120 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.750217 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":72,"thread_start_us":142,"threads_started":1}
I20260808 04:45:07.750432 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.746412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:43355 (local address 127.4.217.1:54371)
0808 04:45:07.746823 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.746857 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.746959 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.747253 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.747255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.747271 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:07.747404 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.747410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.748156 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.748159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.749118 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.749129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.749962 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.749965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.750057 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.750076 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.750225 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.750287 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":42,"thread_start_us":123,"threads_started":1}
I20260808 04:45:07.750439 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.746534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:54371 (local address 127.4.217.3:43355)
0808 04:45:07.746710 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.746717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.746947 (+ 230us) server_negotiation.cc:407] Connection header received
0808 04:45:07.747088 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.747091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.747142 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.747224 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:07.747522 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.748030 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.749275 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.749692 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.749950 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.750016 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.750242 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.750319 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 04:45:07.751199 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7054cb2f12284db5b00db1e38feb8d11" candidate_uuid: "c805e12ed3304184bb93c3b5453d06ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271" is_pre_election: true
I20260808 04:45:07.751590 8479 raft_consensus.cc:2454] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c805e12ed3304184bb93c3b5453d06ec in term 0.
I20260808 04:45:07.751593 8266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7054cb2f12284db5b00db1e38feb8d11" candidate_uuid: "c805e12ed3304184bb93c3b5453d06ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" is_pre_election: true
I20260808 04:45:07.751869 8266 raft_consensus.cc:2454] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c805e12ed3304184bb93c3b5453d06ec in term 0.
I20260808 04:45:07.752116 7923 leader_election.cc:302] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c805e12ed3304184bb93c3b5453d06ec, d033aeb702704f30b07e382d474d3271; no voters:
I20260808 04:45:07.752409 8605 raft_consensus.cc:2781] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:07.752488 8605 raft_consensus.cc:482] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:07.752521 8605 raft_consensus.cc:3037] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:07.753679 8605 raft_consensus.cc:504] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.754081 8605 leader_election.cc:288] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [CANDIDATE]: Term 1 election: Requested vote from peers f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2:44853), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:07.754403 8266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7054cb2f12284db5b00db1e38feb8d11" candidate_uuid: "c805e12ed3304184bb93c3b5453d06ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43"
I20260808 04:45:07.754505 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7054cb2f12284db5b00db1e38feb8d11" candidate_uuid: "c805e12ed3304184bb93c3b5453d06ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271"
I20260808 04:45:07.754520 8266 raft_consensus.cc:3037] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:07.754637 8479 raft_consensus.cc:3037] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:07.755587 8266 raft_consensus.cc:2454] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c805e12ed3304184bb93c3b5453d06ec in term 1.
I20260808 04:45:07.755900 7924 leader_election.cc:302] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c805e12ed3304184bb93c3b5453d06ec, f8619b1fc8cf4fd4b196b01da15a5b43; no voters:
I20260808 04:45:07.756068 8605 raft_consensus.cc:2781] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:07.756300 8605 raft_consensus.cc:686] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 1 LEADER]: Becoming Leader. State: Replica: c805e12ed3304184bb93c3b5453d06ec, State: Running, Role: LEADER
I20260808 04:45:07.756354 8479 raft_consensus.cc:2454] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c805e12ed3304184bb93c3b5453d06ec in term 1.
I20260808 04:45:07.756656 8605 consensus_queue.cc:237] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.760138 7776 catalog_manager.cc:5495] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec reported cstate change: term changed from 0 to 1, leader changed from <none> to c805e12ed3304184bb93c3b5453d06ec (127.4.217.1). New cstate: current_term: 1 leader_uuid: "c805e12ed3304184bb93c3b5453d06ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } health_report { overall_health: UNKNOWN } } }
Aug 08 04:45:07 dist-test-slave-2kcd krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:07.834734 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.826671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.4.217.62:35989)
0808 04:45:07.826824 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.826831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.827019 (+ 188us) server_negotiation.cc:407] Connection header received
0808 04:45:07.827127 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.827129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.827168 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.827247 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:07.827884 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.828387 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.829161 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.829364 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.829440 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.831670 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:07.831674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:07.831676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:07.831697 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:07.833166 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.833621 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.833623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.833625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.833696 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:07.833951 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:07.833953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:07.833954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:07.834142 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:07.834220 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.834490 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.834581 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:45:07.834739 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.826574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35989 (local address 127.0.0.1:33500)
0808 04:45:07.826921 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.826940 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.827049 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.827286 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.827289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.827686 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:07.827783 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.827789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.828480 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.828482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.829063 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.829072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.829461 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.829464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.829526 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.830146 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:07.830158 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:07.831535 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:07.833276 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.833288 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.833296 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.833528 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.833777 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:07.833779 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:07.833780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:07.833855 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:07.834235 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:07.834240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:07.834337 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.834428 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.834534 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
I20260808 04:45:07.838522 7776 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:33500:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:45:07.839196 7776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:07.844044 8000 tablet_service.cc:1467] Processing CreateTablet for tablet 7970ddf7482446258381dacbdf1343e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f90c6730cc954ce0bcc5b0469355a266]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:45:07.844031 8237 tablet_service.cc:1467] Processing CreateTablet for tablet 7970ddf7482446258381dacbdf1343e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f90c6730cc954ce0bcc5b0469355a266]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:45:07.844358 8464 tablet_service.cc:1467] Processing CreateTablet for tablet 7970ddf7482446258381dacbdf1343e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f90c6730cc954ce0bcc5b0469355a266]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:45:07.844859 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7970ddf7482446258381dacbdf1343e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:07.845012 8000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7970ddf7482446258381dacbdf1343e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:07.845079 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7970ddf7482446258381dacbdf1343e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:07.847482 8602 tablet_bootstrap.cc:492] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: Bootstrap starting.
I20260808 04:45:07.847901 8598 tablet_bootstrap.cc:492] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: Bootstrap starting.
I20260808 04:45:07.848594 8602 tablet_bootstrap.cc:654] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:07.848732 8597 tablet_bootstrap.cc:492] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: Bootstrap starting.
I20260808 04:45:07.848961 8598 tablet_bootstrap.cc:654] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:07.850019 8597 tablet_bootstrap.cc:654] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:07.850298 8598 tablet_bootstrap.cc:492] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: No bootstrap required, opened a new log
I20260808 04:45:07.850380 8598 ts_tablet_manager.cc:1397] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:07.850962 8598 raft_consensus.cc:348] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.851091 8598 raft_consensus.cc:374] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:07.851128 8598 raft_consensus.cc:729] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8619b1fc8cf4fd4b196b01da15a5b43, State: Initialized, Role: FOLLOWER
I20260808 04:45:07.851140 8602 tablet_bootstrap.cc:492] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: No bootstrap required, opened a new log
I20260808 04:45:07.851222 8602 ts_tablet_manager.cc:1397] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 04:45:07.851265 8598 consensus_queue.cc:260] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.851435 8597 tablet_bootstrap.cc:492] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: No bootstrap required, opened a new log
I20260808 04:45:07.851560 8597 ts_tablet_manager.cc:1397] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:07.851562 8598 ts_tablet_manager.cc:1428] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:07.851680 8602 raft_consensus.cc:348] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.851797 8602 raft_consensus.cc:374] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:07.851846 8602 raft_consensus.cc:729] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d033aeb702704f30b07e382d474d3271, State: Initialized, Role: FOLLOWER
I20260808 04:45:07.851812 8606 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.851956 8606 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:07.851974 8602 consensus_queue.cc:260] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.852118 8597 raft_consensus.cc:348] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.852237 8602 ts_tablet_manager.cc:1428] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:45:07.852252 8597 raft_consensus.cc:374] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:07.852293 8597 raft_consensus.cc:729] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c805e12ed3304184bb93c3b5453d06ec, State: Initialized, Role: FOLLOWER
I20260808 04:45:07.852432 8597 consensus_queue.cc:260] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.852571 8608 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.852708 8608 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:07.852716 8605 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.852873 8605 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:07.853515 8597 ts_tablet_manager.cc:1428] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:07.874202 8606 raft_consensus.cc:482] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:07.874313 8606 raft_consensus.cc:504] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.875335 8606 leader_election.cc:288] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:07.879571 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.875650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:36989 (local address 127.4.217.1:34939)
0808 04:45:07.875782 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.875785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.875972 (+ 187us) server_negotiation.cc:407] Connection header received
0808 04:45:07.876328 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.876332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.876400 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.876508 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:07.876832 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.877337 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.878463 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.878976 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.879102 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.879170 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.879353 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.879422 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:45:07.879595 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.875602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:34939 (local address 127.4.217.2:36989)
0808 04:45:07.875876 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.875894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.875998 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.876517 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.876520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.876539 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:07.876676 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.876682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.877455 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.877459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.878326 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.878336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.879115 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.879119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.879228 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.879256 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.879336 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.879407 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":32,"thread_start_us":114,"threads_started":1}
I20260808 04:45:07.880623 8036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7970ddf7482446258381dacbdf1343e4" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c805e12ed3304184bb93c3b5453d06ec" is_pre_election: true
I20260808 04:45:07.880806 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.876139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:39239 (local address 127.4.217.3:43355)
0808 04:45:07.876253 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.876257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.876347 (+ 90us) server_negotiation.cc:407] Connection header received
0808 04:45:07.876493 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.876496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.876541 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.876628 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:07.877320 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.877803 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.879072 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.879688 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.879947 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.880015 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.880563 (+ 548us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.880663 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:45:07.880760 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.875998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:43355 (local address 127.4.217.2:39239)
0808 04:45:07.876254 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.876265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.876383 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.876641 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.876644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.876662 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:07.877195 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.877203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.877915 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.877920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.878931 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.878943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.880264 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.880268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.880371 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.880397 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.880525 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.880601 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":34,"thread_start_us":69,"threads_started":1}
I20260808 04:45:07.880934 8036 raft_consensus.cc:2454] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 0.
I20260808 04:45:07.881223 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7970ddf7482446258381dacbdf1343e4" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271" is_pre_election: true
I20260808 04:45:07.881412 8479 raft_consensus.cc:2454] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 0.
I20260808 04:45:07.881438 8156 leader_election.cc:302] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c805e12ed3304184bb93c3b5453d06ec, f8619b1fc8cf4fd4b196b01da15a5b43; no voters:
I20260808 04:45:07.881731 8606 raft_consensus.cc:2781] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:07.881820 8606 raft_consensus.cc:482] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:07.881868 8606 raft_consensus.cc:3037] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:07.882917 8606 raft_consensus.cc:504] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.883256 8606 leader_election.cc:288] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 election: Requested vote from peers c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:07.883615 8036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7970ddf7482446258381dacbdf1343e4" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c805e12ed3304184bb93c3b5453d06ec"
I20260808 04:45:07.883697 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7970ddf7482446258381dacbdf1343e4" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271"
I20260808 04:45:07.883735 8036 raft_consensus.cc:3037] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:07.883810 8479 raft_consensus.cc:3037] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:07.884922 8036 raft_consensus.cc:2454] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 1.
I20260808 04:45:07.884922 8479 raft_consensus.cc:2454] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 1.
I20260808 04:45:07.885206 8156 leader_election.cc:302] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c805e12ed3304184bb93c3b5453d06ec, f8619b1fc8cf4fd4b196b01da15a5b43; no voters:
I20260808 04:45:07.885425 8606 raft_consensus.cc:2781] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:07.885660 8606 raft_consensus.cc:686] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 LEADER]: Becoming Leader. State: Replica: f8619b1fc8cf4fd4b196b01da15a5b43, State: Running, Role: LEADER
I20260808 04:45:07.885977 8606 consensus_queue.cc:237] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:07.888108 8638 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: TxnStatusTablet state changed. Reason: New leader f8619b1fc8cf4fd4b196b01da15a5b43. Latest consensus state: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.888224 8638 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:45:07.888522 8643 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:45:07.888747 8643 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:45:07.889495 7776 catalog_manager.cc:5495] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:07.909642 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.905672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:44853 (local address 127.0.0.1:42984)
0808 04:45:07.905856 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.905872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.905991 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.906386 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.906390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.906398 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:07.906523 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.906529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.907293 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.907302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.907970 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.907982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.908474 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.908478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.908593 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.908642 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:07.909336 (+ 694us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:07.909354 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.909428 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.909513 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:45:07.909665 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.905769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.4.217.2:44853)
0808 04:45:07.905894 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.905901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.905916 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:07.906184 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.906187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.906256 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.906357 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:07.906637 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.907170 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.908130 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.908352 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.908463 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.908813 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:07.909232 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:07.909362 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.909437 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.909491 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:45:07.915537 8479 raft_consensus.cc:1261] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Refusing update from remote peer f8619b1fc8cf4fd4b196b01da15a5b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:07.915537 8036 raft_consensus.cc:1261] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Refusing update from remote peer f8619b1fc8cf4fd4b196b01da15a5b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:07.916363 8638 consensus_queue.cc:1035] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:07.916579 8606 consensus_queue.cc:1035] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:07.920770 8605 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: TxnStatusTablet state changed. Reason: New leader f8619b1fc8cf4fd4b196b01da15a5b43. Latest consensus state: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.920918 8605 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:07.923055 8608 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: TxnStatusTablet state changed. Reason: New leader f8619b1fc8cf4fd4b196b01da15a5b43. Latest consensus state: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.923182 8608 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:07.925717 8606 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.925742 8608 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.925833 8606 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:45:07.925861 8608 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P d033aeb702704f30b07e382d474d3271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:07.931828 8647 mvcc.cc:205] Tried to move back new op lower bound from 7316129005214887936 to 7316129005103411200. Current Snapshot: MvccSnapshot[applied={T|T < 7316129005214887936 or (T in {7316129005214887936})}]
I20260808 04:45:07.931926 8648 mvcc.cc:205] Tried to move back new op lower bound from 7316129005214887936 to 7316129005103411200. Current Snapshot: MvccSnapshot[applied={T|T < 7316129005214887936}]
I20260808 04:45:07.937269 8638 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.937392 8638 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P f8619b1fc8cf4fd4b196b01da15a5b43: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:45:07.938874 8605 tablet_replica.cc:439] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } } }
I20260808 04:45:07.939000 8605 tablet_replica.cc:442] T 7970ddf7482446258381dacbdf1343e4 P c805e12ed3304184bb93c3b5453d06ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:07.939098 8649 mvcc.cc:205] Tried to move back new op lower bound from 7316129005214887936 to 7316129005103411200. Current Snapshot: MvccSnapshot[applied={T|T < 7316129005214887936}]
I20260808 04:45:07.940044 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.935019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.4.217.1:34939)
0808 04:45:07.935148 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.935153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.935431 (+ 278us) server_negotiation.cc:407] Connection header received
0808 04:45:07.935587 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.935591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.935650 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.935758 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:07.936078 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.936867 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.937993 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.938288 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.938382 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.939027 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:07.939552 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:07.939638 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.939832 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.939896 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":52,"spinlock_wait_cycles":896}
I20260808 04:45:07.951789 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.947804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:44853 (local address 127.4.217.1:45953)
0808 04:45:07.948216 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.948228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.948316 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.948572 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.948574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.948581 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:07.948687 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.948692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.949736 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.949738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.950493 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.950512 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.951090 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.951093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.951168 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.951214 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:07.951541 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:07.951546 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.951596 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.951661 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":121,"threads_started":1}
I20260808 04:45:07.951818 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.947902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:45953 (local address 127.4.217.2:44853)
0808 04:45:07.948045 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.948050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.948306 (+ 256us) server_negotiation.cc:407] Connection header received
0808 04:45:07.948405 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.948408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.948463 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.948547 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:07.948812 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.949615 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.950615 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.950953 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.951085 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.951368 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:07.951451 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:07.951525 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.951632 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.951685 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:45:07.959350 8479 raft_consensus.cc:1261] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Refusing update from remote peer c805e12ed3304184bb93c3b5453d06ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:07.959466 8266 raft_consensus.cc:1261] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Refusing update from remote peer c805e12ed3304184bb93c3b5453d06ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:07.959890 8614 consensus_queue.cc:1035] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:07.960269 8614 consensus_queue.cc:1035] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:07.984010 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.980198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:46355 (local address 127.4.217.1:34939)
0808 04:45:07.980349 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:07.980355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:07.980549 (+ 194us) server_negotiation.cc:407] Connection header received
0808 04:45:07.980710 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:07.980713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:07.980767 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:07.980881 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:07.981275 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.981779 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.982926 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.983157 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.983267 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.983563 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:07.983641 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:07.983723 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:07.983815 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 04:45:07.983865 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:45:07.984022 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:07.980137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:34939 (local address 127.4.217.2:46355)
0808 04:45:07.980467 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:07.980482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:07.980617 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:07.980893 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:07.980897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:07.980905 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:07.981036 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.981042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.981892 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.981898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.982795 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:07.982805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:07.983270 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:07.983274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:07.983368 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:07.983421 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:07.983729 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:07.983738 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:45:07.983805 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 04:45:07.983881 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
I20260808 04:45:08.008204 7776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:08.008407 7776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:08.011551 7776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8: Sending DeleteTablet for 3 replicas of tablet 7054cb2f12284db5b00db1e38feb8d11
I20260808 04:45:08.012385 8464 tablet_service.cc:1514] Processing DeleteTablet for tablet 7054cb2f12284db5b00db1e38feb8d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:08 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:55110
I20260808 04:45:08.013132 8000 tablet_service.cc:1514] Processing DeleteTablet for tablet 7054cb2f12284db5b00db1e38feb8d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:08 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:51294
I20260808 04:45:08.013347 8237 tablet_service.cc:1514] Processing DeleteTablet for tablet 7054cb2f12284db5b00db1e38feb8d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:08 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:42970
I20260808 04:45:08.014040 8674 tablet_replica.cc:330] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: stopping tablet replica
I20260808 04:45:08.014400 8674 raft_consensus.cc:2229] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:08.014328 7776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:45:08.014643 8674 raft_consensus.cc:2258] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 04:45:08.014744 7776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:08.016036 8674 ts_tablet_manager.cc:1905] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:08.019145 8674 ts_tablet_manager.cc:1918] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:08.019240 8674 log.cc:1196] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/7054cb2f12284db5b00db1e38feb8d11
I20260808 04:45:08.019577 8674 ts_tablet_manager.cc:1939] T 7054cb2f12284db5b00db1e38feb8d11 P d033aeb702704f30b07e382d474d3271: Deleting consensus metadata
I20260808 04:45:08.020283 8000 tablet_service.cc:1467] Processing CreateTablet for tablet c75ee0348eff45449b0fbf156a5642f9 (DEFAULT_TABLE table=test-table [id=c1f7771f47e548b58cda60938ce0f5ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:08.020586 8000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75ee0348eff45449b0fbf156a5642f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:08.020808 7752 catalog_manager.cc:4841] TS d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355): tablet 7054cb2f12284db5b00db1e38feb8d11 (table test-table [id=7ff409d9618544c9a45a57af62f4f88a]) successfully deleted
I20260808 04:45:08.021128 8679 tablet_replica.cc:330] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: stopping tablet replica
I20260808 04:45:08.021421 8679 raft_consensus.cc:2229] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:08.021575 8679 raft_consensus.cc:2258] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:08.022576 8464 tablet_service.cc:1467] Processing CreateTablet for tablet c75ee0348eff45449b0fbf156a5642f9 (DEFAULT_TABLE table=test-table [id=c1f7771f47e548b58cda60938ce0f5ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:08.022781 8679 ts_tablet_manager.cc:1905] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:08.022820 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75ee0348eff45449b0fbf156a5642f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:08.025151 8602 tablet_bootstrap.cc:492] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: Bootstrap starting.
I20260808 04:45:08.025667 8597 tablet_bootstrap.cc:492] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: Bootstrap starting.
I20260808 04:45:08.026088 8602 tablet_bootstrap.cc:654] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:08.026684 8597 tablet_bootstrap.cc:654] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:08.026863 8677 tablet_replica.cc:330] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: stopping tablet replica
I20260808 04:45:08.027150 8677 raft_consensus.cc:2229] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:08.027520 8677 raft_consensus.cc:2258] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:08.027734 8679 ts_tablet_manager.cc:1918] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:08.027838 8679 log.cc:1196] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/7054cb2f12284db5b00db1e38feb8d11
I20260808 04:45:08.027976 8237 tablet_service.cc:1467] Processing CreateTablet for tablet c75ee0348eff45449b0fbf156a5642f9 (DEFAULT_TABLE table=test-table [id=c1f7771f47e548b58cda60938ce0f5ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:08.028961 8597 tablet_bootstrap.cc:492] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: No bootstrap required, opened a new log
I20260808 04:45:08.029048 8597 ts_tablet_manager.cc:1397] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:08.029531 8597 raft_consensus.cc:348] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:08.029662 8597 raft_consensus.cc:374] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:08.029702 8597 raft_consensus.cc:729] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c805e12ed3304184bb93c3b5453d06ec, State: Initialized, Role: FOLLOWER
I20260808 04:45:08.029623 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75ee0348eff45449b0fbf156a5642f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:08.029829 8597 consensus_queue.cc:260] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:08.030088 8597 ts_tablet_manager.cc:1428] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:08.028180 8679 ts_tablet_manager.cc:1939] T 7054cb2f12284db5b00db1e38feb8d11 P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting consensus metadata
I20260808 04:45:08.032183 7753 catalog_manager.cc:4841] TS f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2:44853): tablet 7054cb2f12284db5b00db1e38feb8d11 (table test-table [id=7ff409d9618544c9a45a57af62f4f88a]) successfully deleted
I20260808 04:45:08.033524 8677 ts_tablet_manager.cc:1905] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:08.034253 8598 tablet_bootstrap.cc:492] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: Bootstrap starting.
I20260808 04:45:08.034379 8602 tablet_bootstrap.cc:492] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: No bootstrap required, opened a new log
I20260808 04:45:08.034466 8602 ts_tablet_manager.cc:1397] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260808 04:45:08.034930 8602 raft_consensus.cc:348] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:08.035048 8602 raft_consensus.cc:374] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:08.035090 8602 raft_consensus.cc:729] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d033aeb702704f30b07e382d474d3271, State: Initialized, Role: FOLLOWER
I20260808 04:45:08.035224 8602 consensus_queue.cc:260] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:08.035480 8602 ts_tablet_manager.cc:1428] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:08.035737 8598 tablet_bootstrap.cc:654] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:08.036602 8677 ts_tablet_manager.cc:1918] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:08.036701 8677 log.cc:1196] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/7054cb2f12284db5b00db1e38feb8d11
I20260808 04:45:08.037155 8677 ts_tablet_manager.cc:1939] T 7054cb2f12284db5b00db1e38feb8d11 P c805e12ed3304184bb93c3b5453d06ec: Deleting consensus metadata
I20260808 04:45:08.038254 7752 catalog_manager.cc:4841] TS c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939): tablet 7054cb2f12284db5b00db1e38feb8d11 (table test-table [id=7ff409d9618544c9a45a57af62f4f88a]) successfully deleted
I20260808 04:45:08.038537 8598 tablet_bootstrap.cc:492] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: No bootstrap required, opened a new log
I20260808 04:45:08.038626 8598 ts_tablet_manager.cc:1397] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 04:45:08.039049 8598 raft_consensus.cc:348] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:08.039162 8598 raft_consensus.cc:374] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:08.039196 8598 raft_consensus.cc:729] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8619b1fc8cf4fd4b196b01da15a5b43, State: Initialized, Role: FOLLOWER
I20260808 04:45:08.039309 8598 consensus_queue.cc:260] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:08.039575 8598 ts_tablet_manager.cc:1428] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:08.190280 8660 raft_consensus.cc:482] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:08.190404 8660 raft_consensus.cc:504] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:08.190747 8660 leader_election.cc:288] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:08.191277 8036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75ee0348eff45449b0fbf156a5642f9" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c805e12ed3304184bb93c3b5453d06ec" is_pre_election: true
I20260808 04:45:08.191277 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75ee0348eff45449b0fbf156a5642f9" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271" is_pre_election: true
I20260808 04:45:08.191439 8036 raft_consensus.cc:2454] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 0.
I20260808 04:45:08.191439 8479 raft_consensus.cc:2454] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 0.
I20260808 04:45:08.191774 8156 leader_election.cc:302] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d033aeb702704f30b07e382d474d3271, f8619b1fc8cf4fd4b196b01da15a5b43; no voters:
I20260808 04:45:08.192001 8660 raft_consensus.cc:2781] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:08.192085 8660 raft_consensus.cc:482] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:08.192112 8660 raft_consensus.cc:3037] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:08.193159 8660 raft_consensus.cc:504] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:08.193490 8660 leader_election.cc:288] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 election: Requested vote from peers c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:08.193858 8036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75ee0348eff45449b0fbf156a5642f9" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c805e12ed3304184bb93c3b5453d06ec"
I20260808 04:45:08.193919 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75ee0348eff45449b0fbf156a5642f9" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271"
I20260808 04:45:08.194001 8036 raft_consensus.cc:3037] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:08.194049 8479 raft_consensus.cc:3037] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:08.194923 8036 raft_consensus.cc:2454] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 1.
I20260808 04:45:08.195173 8479 raft_consensus.cc:2454] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 1.
I20260808 04:45:08.195232 8156 leader_election.cc:302] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c805e12ed3304184bb93c3b5453d06ec, f8619b1fc8cf4fd4b196b01da15a5b43; no voters:
I20260808 04:45:08.195448 8660 raft_consensus.cc:2781] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:08.195574 8660 raft_consensus.cc:686] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 LEADER]: Becoming Leader. State: Replica: f8619b1fc8cf4fd4b196b01da15a5b43, State: Running, Role: LEADER
I20260808 04:45:08.195722 8660 consensus_queue.cc:237] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:08.197465 7776 catalog_manager.cc:5495] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:08.230050 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:08.226861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.4.217.2:44853)
0808 04:45:08.226995 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:08.227002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:08.227025 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:45:08.227194 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:08.227198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:08.227249 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:08.227332 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:08.227597 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:08.228113 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:08.228988 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:08.229212 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:08.229321 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:08.229599 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:08.229672 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:08.229752 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:08.229853 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 04:45:08.229900 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:45:08.238283 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:08.235699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:53381 (local address 127.4.217.2:44853)
0808 04:45:08.235818 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:08.235821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:08.235833 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:45:08.235923 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:08.235926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:08.235963 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:08.236024 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:08.236222 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:08.236649 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:08.237464 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:08.237668 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:08.237743 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:08.237942 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:08.237998 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:08.238053 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:08.238134 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 04:45:08.238169 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:45:08.238298 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:08.235602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:44853 (local address 127.4.217.2:53381)
0808 04:45:08.235758 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:08.235772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:08.235846 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:08.236059 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:08.236061 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:08.236067 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:08.236139 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:08.236143 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:08.236735 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:08.236737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:08.237366 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:08.237374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:08.237766 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:08.237768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:08.237825 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:08.237849 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:08.238075 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:08.238080 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:45:08.238127 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 04:45:08.238188 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:45:08.243965 8036 raft_consensus.cc:1261] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Refusing update from remote peer f8619b1fc8cf4fd4b196b01da15a5b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:08.243964 8479 raft_consensus.cc:1261] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Refusing update from remote peer f8619b1fc8cf4fd4b196b01da15a5b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:08.244355 8660 consensus_queue.cc:1035] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:08.244500 8665 consensus_queue.cc:1035] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:45:08.261323 8647 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 04:45:09.816815 7776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:09.816963 7776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:09.819360 7776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8: Sending DeleteTablet for 3 replicas of tablet c75ee0348eff45449b0fbf156a5642f9
I20260808 04:45:09.819967 8237 tablet_service.cc:1514] Processing DeleteTablet for tablet c75ee0348eff45449b0fbf156a5642f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:09 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:42970
I20260808 04:45:09.820040 8464 tablet_service.cc:1514] Processing DeleteTablet for tablet c75ee0348eff45449b0fbf156a5642f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:09 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:55110
I20260808 04:45:09.820287 8000 tablet_service.cc:1514] Processing DeleteTablet for tablet c75ee0348eff45449b0fbf156a5642f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:09 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:51294
I20260808 04:45:09.820371 8726 tablet_replica.cc:330] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: stopping tablet replica
I20260808 04:45:09.820515 8726 raft_consensus.cc:2229] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:09.820890 8726 raft_consensus.cc:2258] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:09.821166 8727 tablet_replica.cc:330] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: stopping tablet replica
I20260808 04:45:09.821307 8727 raft_consensus.cc:2229] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:09.821430 8727 raft_consensus.cc:2258] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:09.821547 8726 ts_tablet_manager.cc:1905] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:09.824047 8727 ts_tablet_manager.cc:1905] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:09.824350 8728 tablet_replica.cc:330] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: stopping tablet replica
I20260808 04:45:09.824498 8728 raft_consensus.cc:2229] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:09.824591 8726 ts_tablet_manager.cc:1918] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:45:09.824669 8726 log.cc:1196] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/c75ee0348eff45449b0fbf156a5642f9
I20260808 04:45:09.824635 8728 raft_consensus.cc:2258] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:09.825006 8726 ts_tablet_manager.cc:1939] T c75ee0348eff45449b0fbf156a5642f9 P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting consensus metadata
I20260808 04:45:09.825713 7753 catalog_manager.cc:4841] TS f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2:44853): tablet c75ee0348eff45449b0fbf156a5642f9 (table test-table [id=c1f7771f47e548b58cda60938ce0f5ad]) successfully deleted
I20260808 04:45:09.826719 8727 ts_tablet_manager.cc:1918] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:45:09.826802 8727 log.cc:1196] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/c75ee0348eff45449b0fbf156a5642f9
I20260808 04:45:09.827080 8727 ts_tablet_manager.cc:1939] T c75ee0348eff45449b0fbf156a5642f9 P d033aeb702704f30b07e382d474d3271: Deleting consensus metadata
I20260808 04:45:09.827426 8728 ts_tablet_manager.cc:1905] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:09.827740 7752 catalog_manager.cc:4841] TS d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355): tablet c75ee0348eff45449b0fbf156a5642f9 (table test-table [id=c1f7771f47e548b58cda60938ce0f5ad]) successfully deleted
I20260808 04:45:09.830431 8728 ts_tablet_manager.cc:1918] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:45:09.830531 8728 log.cc:1196] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/c75ee0348eff45449b0fbf156a5642f9
I20260808 04:45:09.830772 8728 ts_tablet_manager.cc:1939] T c75ee0348eff45449b0fbf156a5642f9 P c805e12ed3304184bb93c3b5453d06ec: Deleting consensus metadata
I20260808 04:45:09.831391 7752 catalog_manager.cc:4841] TS c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939): tablet c75ee0348eff45449b0fbf156a5642f9 (table test-table [id=c1f7771f47e548b58cda60938ce0f5ad]) successfully deleted
Aug 08 04:45:09 dist-test-slave-2kcd krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:45:09 dist-test-slave-2kcd krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:09.849886 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:09.841137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.4.217.62:35989)
0808 04:45:09.841412 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:09.841422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:09.841523 (+ 101us) server_negotiation.cc:407] Connection header received
0808 04:45:09.841693 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:09.841696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:09.841743 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:09.841834 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:09.842571 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:09.843123 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:09.843977 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:09.844215 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:09.844325 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:09.846722 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:09.846726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:09.846728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:09.846752 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:09.848049 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:09.848469 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:09.848473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:09.848475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:09.848588 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:09.848903 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:09.848906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:09.848908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:09.849095 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:09.849244 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:09.849514 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 04:45:09.849645 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260808 04:45:09.852387 7776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:09.852900 7776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:09.857403 8000 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2e962317bd4f66bbff54b2d42067df (DEFAULT_TABLE table=test-table [id=b4f6f5a8733d442ba6ce3b5ee4030b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:09.857502 8237 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2e962317bd4f66bbff54b2d42067df (DEFAULT_TABLE table=test-table [id=b4f6f5a8733d442ba6ce3b5ee4030b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:09.857729 8000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2e962317bd4f66bbff54b2d42067df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:09.857685 8464 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2e962317bd4f66bbff54b2d42067df (DEFAULT_TABLE table=test-table [id=b4f6f5a8733d442ba6ce3b5ee4030b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:09.857795 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2e962317bd4f66bbff54b2d42067df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:09.857945 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2e962317bd4f66bbff54b2d42067df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:09.859928 8741 tablet_bootstrap.cc:492] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: Bootstrap starting.
I20260808 04:45:09.860321 8744 tablet_bootstrap.cc:492] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: Bootstrap starting.
I20260808 04:45:09.860986 8741 tablet_bootstrap.cc:654] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:09.861023 8742 tablet_bootstrap.cc:492] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: Bootstrap starting.
I20260808 04:45:09.861375 8744 tablet_bootstrap.cc:654] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:09.862201 8741 tablet_bootstrap.cc:492] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: No bootstrap required, opened a new log
I20260808 04:45:09.862277 8741 ts_tablet_manager.cc:1397] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 04:45:09.862511 8742 tablet_bootstrap.cc:654] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:09.863036 8741 raft_consensus.cc:348] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.863180 8741 raft_consensus.cc:374] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:09.863232 8741 raft_consensus.cc:729] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c805e12ed3304184bb93c3b5453d06ec, State: Initialized, Role: FOLLOWER
I20260808 04:45:09.863315 8744 tablet_bootstrap.cc:492] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: No bootstrap required, opened a new log
I20260808 04:45:09.863420 8744 ts_tablet_manager.cc:1397] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 04:45:09.863411 8741 consensus_queue.cc:260] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.863790 8741 ts_tablet_manager.cc:1428] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:45:09.863906 8744 raft_consensus.cc:348] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.864029 8744 raft_consensus.cc:374] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:09.864104 8744 raft_consensus.cc:729] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8619b1fc8cf4fd4b196b01da15a5b43, State: Initialized, Role: FOLLOWER
I20260808 04:45:09.864248 8744 consensus_queue.cc:260] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.864576 8744 ts_tablet_manager.cc:1428] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:45:09.865444 8742 tablet_bootstrap.cc:492] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: No bootstrap required, opened a new log
I20260808 04:45:09.865520 8742 ts_tablet_manager.cc:1397] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 04:45:09.865998 8742 raft_consensus.cc:348] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.866134 8742 raft_consensus.cc:374] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:09.866176 8742 raft_consensus.cc:729] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d033aeb702704f30b07e382d474d3271, State: Initialized, Role: FOLLOWER
I20260808 04:45:09.866308 8742 consensus_queue.cc:260] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.866729 8742 ts_tablet_manager.cc:1428] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:09.901875 8752 raft_consensus.cc:482] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:09.902021 8752 raft_consensus.cc:504] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.902490 8752 leader_election.cc:288] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2:44853), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:09.902920 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e962317bd4f66bbff54b2d42067df" candidate_uuid: "c805e12ed3304184bb93c3b5453d06ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271" is_pre_election: true
I20260808 04:45:09.902928 8266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e962317bd4f66bbff54b2d42067df" candidate_uuid: "c805e12ed3304184bb93c3b5453d06ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" is_pre_election: true
I20260808 04:45:09.903087 8479 raft_consensus.cc:2454] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c805e12ed3304184bb93c3b5453d06ec in term 0.
I20260808 04:45:09.903105 8266 raft_consensus.cc:2454] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c805e12ed3304184bb93c3b5453d06ec in term 0.
I20260808 04:45:09.903470 7924 leader_election.cc:302] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c805e12ed3304184bb93c3b5453d06ec, f8619b1fc8cf4fd4b196b01da15a5b43; no voters:
I20260808 04:45:09.903692 8752 raft_consensus.cc:2781] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:09.903759 8752 raft_consensus.cc:482] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:09.903787 8752 raft_consensus.cc:3037] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:09.904984 8752 raft_consensus.cc:504] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.905346 8752 leader_election.cc:288] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [CANDIDATE]: Term 1 election: Requested vote from peers f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2:44853), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:09.905684 8266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e962317bd4f66bbff54b2d42067df" candidate_uuid: "c805e12ed3304184bb93c3b5453d06ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43"
I20260808 04:45:09.905735 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e962317bd4f66bbff54b2d42067df" candidate_uuid: "c805e12ed3304184bb93c3b5453d06ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271"
I20260808 04:45:09.905800 8266 raft_consensus.cc:3037] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:09.905865 8479 raft_consensus.cc:3037] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:09.906734 8266 raft_consensus.cc:2454] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c805e12ed3304184bb93c3b5453d06ec in term 1.
I20260808 04:45:09.906826 8479 raft_consensus.cc:2454] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c805e12ed3304184bb93c3b5453d06ec in term 1.
I20260808 04:45:09.907024 7924 leader_election.cc:302] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c805e12ed3304184bb93c3b5453d06ec, f8619b1fc8cf4fd4b196b01da15a5b43; no voters:
I20260808 04:45:09.907199 8752 raft_consensus.cc:2781] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:09.907450 8752 raft_consensus.cc:686] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 1 LEADER]: Becoming Leader. State: Replica: c805e12ed3304184bb93c3b5453d06ec, State: Running, Role: LEADER
I20260808 04:45:09.907605 8752 consensus_queue.cc:237] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.909318 7774 catalog_manager.cc:5495] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec reported cstate change: term changed from 0 to 1, leader changed from <none> to c805e12ed3304184bb93c3b5453d06ec (127.4.217.1). New cstate: current_term: 1 leader_uuid: "c805e12ed3304184bb93c3b5453d06ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:09.936170 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:09.932370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.4.217.1:34939)
0808 04:45:09.932702 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:09.932711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:09.932726 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:09.932771 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:09.932773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:09.932892 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:09.932995 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:09.933282 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:09.933990 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:09.935130 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:09.935362 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:09.935452 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:09.935675 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:09.935745 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:09.935801 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:09.935913 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 04:45:09.935988 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":164,"threads_started":1}
I20260808 04:45:09.942342 8266 raft_consensus.cc:1261] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Refusing update from remote peer c805e12ed3304184bb93c3b5453d06ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:09.942680 8479 raft_consensus.cc:1261] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Refusing update from remote peer c805e12ed3304184bb93c3b5453d06ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:09.942814 8752 consensus_queue.cc:1035] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:09.943094 8757 consensus_queue.cc:1035] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:09.973815 7774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:09.973977 7774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:09.975845 7774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8: Sending DeleteTablet for 3 replicas of tablet 9a2e962317bd4f66bbff54b2d42067df
I20260808 04:45:09.976353 8000 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2e962317bd4f66bbff54b2d42067df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:09 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:51294
I20260808 04:45:09.976420 8237 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2e962317bd4f66bbff54b2d42067df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:09 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:42970
I20260808 04:45:09.976428 8464 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2e962317bd4f66bbff54b2d42067df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:09 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:55110
I20260808 04:45:09.976526 8728 tablet_replica.cc:330] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: stopping tablet replica
I20260808 04:45:09.976588 8726 tablet_replica.cc:330] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: stopping tablet replica
I20260808 04:45:09.976598 8727 tablet_replica.cc:330] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: stopping tablet replica
I20260808 04:45:09.976644 8728 raft_consensus.cc:2229] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:09.976684 8726 raft_consensus.cc:2229] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:09.976693 8727 raft_consensus.cc:2229] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:09.976812 8726 raft_consensus.cc:2258] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:09.976836 8728 raft_consensus.cc:2258] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:09.976881 8727 raft_consensus.cc:2258] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:09.977116 7774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:09.977525 7774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:09.977559 8728 ts_tablet_manager.cc:1905] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:09.977564 8727 ts_tablet_manager.cc:1905] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:09.977744 8726 ts_tablet_manager.cc:1905] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:09.980063 8728 ts_tablet_manager.cc:1918] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:09.980054 8727 ts_tablet_manager.cc:1918] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:09.980144 8728 log.cc:1196] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/9a2e962317bd4f66bbff54b2d42067df
I20260808 04:45:09.980175 8727 log.cc:1196] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/9a2e962317bd4f66bbff54b2d42067df
I20260808 04:45:09.980425 8727 ts_tablet_manager.cc:1939] T 9a2e962317bd4f66bbff54b2d42067df P d033aeb702704f30b07e382d474d3271: Deleting consensus metadata
I20260808 04:45:09.980425 8728 ts_tablet_manager.cc:1939] T 9a2e962317bd4f66bbff54b2d42067df P c805e12ed3304184bb93c3b5453d06ec: Deleting consensus metadata
I20260808 04:45:09.980556 8726 ts_tablet_manager.cc:1918] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:09.980626 8726 log.cc:1196] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/9a2e962317bd4f66bbff54b2d42067df
I20260808 04:45:09.980928 8726 ts_tablet_manager.cc:1939] T 9a2e962317bd4f66bbff54b2d42067df P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting consensus metadata
I20260808 04:45:09.981040 7752 catalog_manager.cc:4841] TS c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939): tablet 9a2e962317bd4f66bbff54b2d42067df (table test-table [id=b4f6f5a8733d442ba6ce3b5ee4030b92]) successfully deleted
I20260808 04:45:09.981181 7752 catalog_manager.cc:4841] TS d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355): tablet 9a2e962317bd4f66bbff54b2d42067df (table test-table [id=b4f6f5a8733d442ba6ce3b5ee4030b92]) successfully deleted
I20260808 04:45:09.981537 7753 catalog_manager.cc:4841] TS f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2:44853): tablet 9a2e962317bd4f66bbff54b2d42067df (table test-table [id=b4f6f5a8733d442ba6ce3b5ee4030b92]) successfully deleted
I20260808 04:45:09.982887 8000 tablet_service.cc:1467] Processing CreateTablet for tablet fd10891ef35b4c6bbf25d736d765f1d8 (DEFAULT_TABLE table=test-table [id=017fad4c8f3848b5830e586683c6b654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:09.982898 8464 tablet_service.cc:1467] Processing CreateTablet for tablet fd10891ef35b4c6bbf25d736d765f1d8 (DEFAULT_TABLE table=test-table [id=017fad4c8f3848b5830e586683c6b654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:09.983132 8000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd10891ef35b4c6bbf25d736d765f1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:09.983152 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd10891ef35b4c6bbf25d736d765f1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:09.985113 8741 tablet_bootstrap.cc:492] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec: Bootstrap starting.
I20260808 04:45:09.985231 8742 tablet_bootstrap.cc:492] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: Bootstrap starting.
I20260808 04:45:09.986029 8741 tablet_bootstrap.cc:654] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:09.986068 8742 tablet_bootstrap.cc:654] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:09.986979 8237 tablet_service.cc:1467] Processing CreateTablet for tablet fd10891ef35b4c6bbf25d736d765f1d8 (DEFAULT_TABLE table=test-table [id=017fad4c8f3848b5830e586683c6b654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:09.987243 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd10891ef35b4c6bbf25d736d765f1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:09.987272 8742 tablet_bootstrap.cc:492] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: No bootstrap required, opened a new log
I20260808 04:45:09.987349 8742 ts_tablet_manager.cc:1397] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:45:09.987501 8741 tablet_bootstrap.cc:492] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec: No bootstrap required, opened a new log
I20260808 04:45:09.987591 8741 ts_tablet_manager.cc:1397] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 04:45:09.987814 8742 raft_consensus.cc:348] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.987926 8742 raft_consensus.cc:374] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:09.987972 8742 raft_consensus.cc:729] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d033aeb702704f30b07e382d474d3271, State: Initialized, Role: FOLLOWER
I20260808 04:45:09.987984 8741 raft_consensus.cc:348] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.988091 8741 raft_consensus.cc:374] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:09.988133 8741 raft_consensus.cc:729] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c805e12ed3304184bb93c3b5453d06ec, State: Initialized, Role: FOLLOWER
I20260808 04:45:09.988111 8742 consensus_queue.cc:260] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.988250 8741 consensus_queue.cc:260] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.988379 8742 ts_tablet_manager.cc:1428] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 04:45:09.988504 8741 ts_tablet_manager.cc:1428] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:45:09.989964 8744 tablet_bootstrap.cc:492] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: Bootstrap starting.
I20260808 04:45:09.990787 8744 tablet_bootstrap.cc:654] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:09.991806 8744 tablet_bootstrap.cc:492] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: No bootstrap required, opened a new log
I20260808 04:45:09.991879 8744 ts_tablet_manager.cc:1397] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 04:45:09.992280 8744 raft_consensus.cc:348] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.992419 8744 raft_consensus.cc:374] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:09.992466 8744 raft_consensus.cc:729] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8619b1fc8cf4fd4b196b01da15a5b43, State: Initialized, Role: FOLLOWER
I20260808 04:45:09.992610 8744 consensus_queue.cc:260] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:09.992949 8744 ts_tablet_manager.cc:1428] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:10.006881 8771 raft_consensus.cc:482] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:10.006991 8771 raft_consensus.cc:504] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:10.007383 8771 leader_election.cc:288] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:10.007753 8036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd10891ef35b4c6bbf25d736d765f1d8" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c805e12ed3304184bb93c3b5453d06ec" is_pre_election: true
I20260808 04:45:10.007759 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd10891ef35b4c6bbf25d736d765f1d8" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271" is_pre_election: true
I20260808 04:45:10.007889 8036 raft_consensus.cc:2454] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 0.
I20260808 04:45:10.007889 8479 raft_consensus.cc:2454] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 0.
I20260808 04:45:10.008152 8156 leader_election.cc:302] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c805e12ed3304184bb93c3b5453d06ec, f8619b1fc8cf4fd4b196b01da15a5b43; no voters:
I20260808 04:45:10.008308 8771 raft_consensus.cc:2781] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:10.008368 8771 raft_consensus.cc:482] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:10.008416 8771 raft_consensus.cc:3037] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:10.009395 8771 raft_consensus.cc:504] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:10.009699 8771 leader_election.cc:288] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 election: Requested vote from peers c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939), d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355)
I20260808 04:45:10.010041 8036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd10891ef35b4c6bbf25d736d765f1d8" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c805e12ed3304184bb93c3b5453d06ec"
I20260808 04:45:10.010052 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd10891ef35b4c6bbf25d736d765f1d8" candidate_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033aeb702704f30b07e382d474d3271"
I20260808 04:45:10.010176 8479 raft_consensus.cc:3037] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:10.010176 8036 raft_consensus.cc:3037] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:10.011268 8036 raft_consensus.cc:2454] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 1.
I20260808 04:45:10.011268 8479 raft_consensus.cc:2454] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8619b1fc8cf4fd4b196b01da15a5b43 in term 1.
I20260808 04:45:10.011572 8156 leader_election.cc:302] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c805e12ed3304184bb93c3b5453d06ec, f8619b1fc8cf4fd4b196b01da15a5b43; no voters:
I20260808 04:45:10.011752 8771 raft_consensus.cc:2781] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:10.011845 8771 raft_consensus.cc:686] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 LEADER]: Becoming Leader. State: Replica: f8619b1fc8cf4fd4b196b01da15a5b43, State: Running, Role: LEADER
I20260808 04:45:10.011983 8771 consensus_queue.cc:237] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } }
I20260808 04:45:10.013543 7775 catalog_manager.cc:5495] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8619b1fc8cf4fd4b196b01da15a5b43" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:10.033627 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:10.030278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.4.217.2:44853)
0808 04:45:10.030580 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:10.030590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:10.030604 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:45:10.030645 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:10.030647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:10.030685 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:10.030775 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:10.031033 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:10.031590 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:10.032477 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:10.032712 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:10.032843 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:10.033134 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:10.033206 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:10.033290 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:10.033411 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 04:45:10.033463 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260808 04:45:10.039285 8479 raft_consensus.cc:1261] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Refusing update from remote peer f8619b1fc8cf4fd4b196b01da15a5b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:10.039369 8036 raft_consensus.cc:1261] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec [term 1 FOLLOWER]: Refusing update from remote peer f8619b1fc8cf4fd4b196b01da15a5b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:10.039748 8693 consensus_queue.cc:1035] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d033aeb702704f30b07e382d474d3271" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:10.039932 8771 consensus_queue.cc:1035] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c805e12ed3304184bb93c3b5453d06ec" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:45:10.053169 8691 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 04:45:11.602937 7775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:11.603101 7775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:11.605852 7775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75298fafbc304a92ba126b8cec14a3e8: Sending DeleteTablet for 3 replicas of tablet fd10891ef35b4c6bbf25d736d765f1d8
I20260808 04:45:11.606477 8000 tablet_service.cc:1514] Processing DeleteTablet for tablet fd10891ef35b4c6bbf25d736d765f1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:11 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:51294
I20260808 04:45:11.606573 8235 tablet_service.cc:1514] Processing DeleteTablet for tablet fd10891ef35b4c6bbf25d736d765f1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:11 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:42970
I20260808 04:45:11.606930 8818 tablet_replica.cc:330] T fd10891ef35b4c6bbf25d736d765f1d8 P c805e12ed3304184bb93c3b5453d06ec: stopping tablet replica
I20260808 04:45:11.606866 8464 tablet_service.cc:1514] Processing DeleteTablet for tablet fd10891ef35b4c6bbf25d736d765f1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:11 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:55110
I20260808 04:45:11.606981 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 7892
I20260808 04:45:11.618425 8821 tablet_replica.cc:330] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: stopping tablet replica
I20260808 04:45:11.618649 8821 raft_consensus.cc:2229] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:11.618835 8821 raft_consensus.cc:2258] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:11.619153 8820 tablet_replica.cc:330] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: stopping tablet replica
I20260808 04:45:11.619302 8820 raft_consensus.cc:2229] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:11.619560 8820 raft_consensus.cc:2258] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:11.619658 8821 ts_tablet_manager.cc:1905] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:11.620213 8820 ts_tablet_manager.cc:1905] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:11.622629 8821 ts_tablet_manager.cc:1918] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:45:11.622720 8821 log.cc:1196] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/fd10891ef35b4c6bbf25d736d765f1d8
I20260808 04:45:11.623018 8821 ts_tablet_manager.cc:1939] T fd10891ef35b4c6bbf25d736d765f1d8 P d033aeb702704f30b07e382d474d3271: Deleting consensus metadata
I20260808 04:45:11.623036 8820 ts_tablet_manager.cc:1918] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:45:11.623102 8820 log.cc:1196] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/fd10891ef35b4c6bbf25d736d765f1d8
I20260808 04:45:11.623459 8820 ts_tablet_manager.cc:1939] T fd10891ef35b4c6bbf25d736d765f1d8 P f8619b1fc8cf4fd4b196b01da15a5b43: Deleting consensus metadata
I20260808 04:45:11.623891 7752 catalog_manager.cc:4841] TS d033aeb702704f30b07e382d474d3271 (127.4.217.3:43355): tablet fd10891ef35b4c6bbf25d736d765f1d8 (table test-table [id=017fad4c8f3848b5830e586683c6b654]) successfully deleted
I20260808 04:45:11.624202 7753 catalog_manager.cc:4841] TS f8619b1fc8cf4fd4b196b01da15a5b43 (127.4.217.2:44853): tablet fd10891ef35b4c6bbf25d736d765f1d8 (table test-table [id=017fad4c8f3848b5830e586683c6b654]) successfully deleted
W20260808 04:45:11.627584 7752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:34939 (error 108)
I20260808 04:45:11.627883 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 8125
I20260808 04:45:11.633900 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:11.631261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:34939 (local address 127.0.0.1:48820)
0808 04:45:11.633733 (+ 2472us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:11.633806 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2393,"thread_start_us":1121,"threads_started":1}
W20260808 04:45:11.634109 7752 catalog_manager.cc:4584] TS c805e12ed3304184bb93c3b5453d06ec (127.4.217.1:34939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd10891ef35b4c6bbf25d736d765f1d8: Network error: Client connection negotiation failed: client connection to 127.4.217.1:34939: connect: Connection refused (error 111)
I20260808 04:45:11.642663 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 8355
I20260808 04:45:11.660490 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 7724
2026-08-08T04:45:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6584 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:11 dist-test-slave-2kcd krb5kdc[8826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:11 dist-test-slave-2kcd krb5kdc[8826](info): set up 2 sockets
Aug 08 04:45:11 dist-test-slave-2kcd krb5kdc[8826](info): commencing operation
krb5kdc: starting...
W20260808 04:45:12.917040 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:13 dist-test-slave-2kcd krb5kdc[8826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:13.050058 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:46793
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:46793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:13.162922 8845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:13.163254 8845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:13.163329 8845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:13.167292 8845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:13.167383 8845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:13.167423 8845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:13.167477 8845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:13.167536 8845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:13.172662 8845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:46793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:46793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:13.173702 8845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:13.174721 8845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:13.183490 8861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:13.183650 8858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:13.183689 8856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:13.183743 8845 server_base.cc:1034] running on GCE node
I20260808 04:45:13.184460 8845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:13.185613 8845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:13.186803 8845 hybrid_clock.cc:648] HybridClock initialized: now 1786164313186787 us; error 74 us; skew 500 ppm
Aug 08 04:45:13 dist-test-slave-2kcd krb5kdc[8826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:13.189543 8845 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:13.190551 8845 webserver.cc:460] Webserver started at http://127.4.217.62:44529/ using document root <none> and password file <none>
I20260808 04:45:13.191104 8845 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:13.191196 8845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:13.191432 8845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:13.193476 8845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "2a190c1f4e054edf828ea054cd25d8f6"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "b82f4223de02e2f39543f0360c5db35d"
server_key_iv: "67e9d59deaa0ae7073cd8f3cb8b97e3f"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.194038 8845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "2a190c1f4e054edf828ea054cd25d8f6"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "b82f4223de02e2f39543f0360c5db35d"
server_key_iv: "67e9d59deaa0ae7073cd8f3cb8b97e3f"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.197579 8845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:45:13.199853 8877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:13.201067 8845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:13.201203 8845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "2a190c1f4e054edf828ea054cd25d8f6"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "b82f4223de02e2f39543f0360c5db35d"
server_key_iv: "67e9d59deaa0ae7073cd8f3cb8b97e3f"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.201309 8845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:13.215054 8845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:13.217801 8845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:13.218005 8845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:13.225340 8845 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:46793
I20260808 04:45:13.225405 8970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:46793 every 8 connection(s)
I20260808 04:45:13.226279 8845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:13.226838 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 8845
I20260808 04:45:13.226964 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:13.227234 4964 external_mini_cluster.cc:1428] Setting key 92056809f428c8d9bf69da1c26779977
I20260808 04:45:13.229507 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:13.236505 8971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6: Bootstrap starting.
Aug 08 04:45:13 dist-test-slave-2kcd krb5kdc[8826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:13.239210 8971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:13.240123 8971 log.cc:824] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:13.242121 8971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6: No bootstrap required, opened a new log
I20260808 04:45:13.242137 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:13.228648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.4.217.62:46793)
0808 04:45:13.229109 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:13.229124 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:13.229159 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:45:13.229824 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:13.229842 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:13.230154 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:13.230416 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:13.231420 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.232675 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.233851 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.234318 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.234434 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:13.237393 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:13.237398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:13.237411 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:13.237440 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:13.239088 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:13.239609 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:13.239614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:13.239618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:13.239726 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:13.240038 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:13.240041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:13.240042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:13.240496 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:13.240707 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:13.240980 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 04:45:13.241276 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":185,"threads_started":1}
I20260808 04:45:13.245905 8971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a190c1f4e054edf828ea054cd25d8f6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46793 } }
I20260808 04:45:13.246124 8971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:13.246161 8971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a190c1f4e054edf828ea054cd25d8f6, State: Initialized, Role: FOLLOWER
I20260808 04:45:13.246503 8971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a190c1f4e054edf828ea054cd25d8f6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46793 } }
I20260808 04:45:13.246608 8971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:13.246651 8971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:13.246718 8971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:13.247505 8971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a190c1f4e054edf828ea054cd25d8f6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46793 } }
I20260808 04:45:13.247799 8971 leader_election.cc:302] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a190c1f4e054edf828ea054cd25d8f6; no voters:
I20260808 04:45:13.248070 8971 leader_election.cc:288] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:13.248193 8980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:13.248484 8980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [term 1 LEADER]: Becoming Leader. State: Replica: 2a190c1f4e054edf828ea054cd25d8f6, State: Running, Role: LEADER
I20260808 04:45:13.248888 8980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a190c1f4e054edf828ea054cd25d8f6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46793 } }
I20260808 04:45:13.249398 8971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:13.250680 8983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a190c1f4e054edf828ea054cd25d8f6. Latest consensus state: current_term: 1 leader_uuid: "2a190c1f4e054edf828ea054cd25d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a190c1f4e054edf828ea054cd25d8f6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46793 } } }
I20260808 04:45:13.250820 8983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:13.251240 8994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:13.251201 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a190c1f4e054edf828ea054cd25d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a190c1f4e054edf828ea054cd25d8f6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46793 } } }
I20260808 04:45:13.252012 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:13.254508 8994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:13.260116 8994 catalog_manager.cc:1347] Generated new cluster ID: 1cf2b81b662640dd9dbfc4e184c1063e
I20260808 04:45:13.260177 8994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:13.291639 8994 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:13.292637 8994 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:13.304526 8994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a190c1f4e054edf828ea054cd25d8f6: Generated new TSK 0
I20260808 04:45:13.305393 8994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:45:13.339344 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46793
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:13.456979 9010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:13.457255 9010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:13.457300 9010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:13.461406 9010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:13.461470 9010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:13.461560 9010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:13.467067 9010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:13.468185 9010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:13.469300 9010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:13.479290 9017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:13.479362 9022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:13.479349 9018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:13.479460 9010 server_base.cc:1034] running on GCE node
I20260808 04:45:13.479970 9010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:13.480679 9010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:13.481879 9010 hybrid_clock.cc:648] HybridClock initialized: now 1786164313481872 us; error 37 us; skew 500 ppm
Aug 08 04:45:13 dist-test-slave-2kcd krb5kdc[8826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:13.484592 9010 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:45:13.485496 9010 webserver.cc:460] Webserver started at http://127.4.217.1:42577/ using document root <none> and password file <none>
I20260808 04:45:13.486059 9010 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:13.486119 9010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:13.486284 9010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:13.488263 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "dd59a485399940edb11e5e08f1668021"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "e6022be0328d0606ed62950ad8e0b29c"
server_key_iv: "a536d166af9327feab2955c55893b62a"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.488842 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "dd59a485399940edb11e5e08f1668021"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "e6022be0328d0606ed62950ad8e0b29c"
server_key_iv: "a536d166af9327feab2955c55893b62a"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.492509 9010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:45:13.494959 9033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:13.496235 9010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:13.496343 9010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "dd59a485399940edb11e5e08f1668021"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "e6022be0328d0606ed62950ad8e0b29c"
server_key_iv: "a536d166af9327feab2955c55893b62a"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.496433 9010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:13.525846 9010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:13.528895 9010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:13.529057 9010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:13.529635 9010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:13.530465 9010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:13.530514 9010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:13.530556 9010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:13.530575 9010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:13.539850 9010 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:35191
I20260808 04:45:13.539880 9221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:35191 every 8 connection(s)
I20260808 04:45:13.540750 9010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:13.546900 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9010
I20260808 04:45:13.547037 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:13.547338 4964 external_mini_cluster.cc:1428] Setting key cc2801ca18a72c2cc748bf20f2ca98b6
Aug 08 04:45:13 dist-test-slave-2kcd krb5kdc[8826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:13.555791 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:13.542721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:54563 (local address 127.4.217.62:46793)
0808 04:45:13.542867 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:13.542870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:13.543647 (+ 777us) server_negotiation.cc:407] Connection header received
0808 04:45:13.545090 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:13.545093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:13.545155 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:13.545262 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:13.546427 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.546978 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.548260 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.548502 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.548645 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:13.551794 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:13.551797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:13.551799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:13.551823 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:13.553317 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:13.554048 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:13.554053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:13.554055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:13.554162 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:13.554597 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:13.554599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:13.554600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:13.554774 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:13.554868 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:13.555558 (+ 690us) server_negotiation.cc:299] Negotiation successful
0808 04:45:13.555668 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:45:13.557132 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:13.543010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:46793 (local address 127.4.217.1:54563)
0808 04:45:13.543493 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:13.543535 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:13.544881 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:13.545453 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:13.545460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:13.545810 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:13.546271 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:13.546284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.547153 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.547156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:13.548137 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:13.548147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.548730 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.548734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:13.548951 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:13.549622 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:13.549638 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:13.551608 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:13.553441 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:13.553450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:13.553468 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:13.553902 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:13.554282 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:13.554288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:13.554291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:13.554485 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:13.554896 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:13.554906 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:13.555269 (+ 363us) client_negotiation.cc:769] Sending connection context
0808 04:45:13.555533 (+ 264us) client_negotiation.cc:240] Negotiation successful
0808 04:45:13.555931 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":131,"threads_started":1}
I20260808 04:45:13.558478 9222 heartbeater.cc:347] Connected to a master server at 127.4.217.62:46793
I20260808 04:45:13.558745 9222 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
I20260808 04:45:13.559300 9222 heartbeater.cc:511] Master 127.4.217.62:46793 requested a full tablet report, sending...
I20260808 04:45:13.560945 8897 ts_manager.cc:194] Registered new tserver with Master: dd59a485399940edb11e5e08f1668021 (127.4.217.1:35191)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:45:13.562481 8897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:54563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:45:13.577322 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46793
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:13.691023 9230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:13.691260 9230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:13.691300 9230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:13.695243 9230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:13.695305 9230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:13.695389 9230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:13.700730 9230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:13.701787 9230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:13.702800 9230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:13.712769 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:13.712860 9230 server_base.cc:1034] running on GCE node
W20260808 04:45:13.712749 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:13.712749 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:13.713503 9230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:13.714227 9230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:13.715420 9230 hybrid_clock.cc:648] HybridClock initialized: now 1786164313715377 us; error 52 us; skew 500 ppm
Aug 08 04:45:13 dist-test-slave-2kcd krb5kdc[8826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:13.718253 9230 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:45:13.719223 9230 webserver.cc:460] Webserver started at http://127.4.217.2:35957/ using document root <none> and password file <none>
I20260808 04:45:13.719805 9230 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:13.719901 9230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:13.720208 9230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:13.722458 9230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "58fd9c946b964edc8a350361334122d5"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "f91dc4b64ebf53e0d6f51c97ade012be"
server_key_iv: "0133dac52a054254a339308daab4e010"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.722987 9230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "58fd9c946b964edc8a350361334122d5"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "f91dc4b64ebf53e0d6f51c97ade012be"
server_key_iv: "0133dac52a054254a339308daab4e010"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.726735 9230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 04:45:13.729130 9256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:13.730352 9230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:13.730458 9230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "58fd9c946b964edc8a350361334122d5"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "f91dc4b64ebf53e0d6f51c97ade012be"
server_key_iv: "0133dac52a054254a339308daab4e010"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.730541 9230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:13.742168 9230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:13.744977 9230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:13.745136 9230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:13.745661 9230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:13.746433 9230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:13.746479 9230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:13.746517 9230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:13.746541 9230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:13.756237 9230 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:34291
I20260808 04:45:13.756256 9463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:34291 every 8 connection(s)
I20260808 04:45:13.757160 9230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:13.763234 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9230
I20260808 04:45:13.763327 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:13.763592 4964 external_mini_cluster.cc:1428] Setting key d337ee9c649579cafcdf36bd87ca3894
Aug 08 04:45:13 dist-test-slave-2kcd krb5kdc[8826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:13.771700 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:13.759038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:56841 (local address 127.4.217.62:46793)
0808 04:45:13.759192 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:13.759196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:13.759994 (+ 798us) server_negotiation.cc:407] Connection header received
0808 04:45:13.761454 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:13.761457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:13.761532 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:13.761616 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:13.762786 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.763372 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.764888 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.765218 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.765332 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:13.767997 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:13.768001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:13.768003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:13.768026 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:13.769489 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:13.769998 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:13.770003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:13.770005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:13.770109 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:13.770599 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:13.770602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:13.770603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:13.770774 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:13.770889 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:13.771414 (+ 525us) server_negotiation.cc:299] Negotiation successful
0808 04:45:13.771540 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:45:13.772550 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:13.759391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:46793 (local address 127.4.217.2:56841)
0808 04:45:13.759836 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:13.759878 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:13.761236 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:13.761809 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:13.761816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:13.762168 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:13.762639 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:13.762652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.763490 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.763499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:13.764661 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:13.764672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:13.765375 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:13.765378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:13.765531 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:13.766146 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:13.766162 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:13.767824 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:13.769613 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:13.769619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:13.769633 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:13.769895 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:13.770245 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:13.770251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:13.770256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:13.770465 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:13.770877 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:13.770883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:13.771165 (+ 282us) client_negotiation.cc:769] Sending connection context
0808 04:45:13.771364 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 04:45:13.771699 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260808 04:45:13.773950 9464 heartbeater.cc:347] Connected to a master server at 127.4.217.62:46793
I20260808 04:45:13.774209 9464 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:45:13.774729 9464 heartbeater.cc:511] Master 127.4.217.62:46793 requested a full tablet report, sending...
I20260808 04:45:13.775902 8897 ts_manager.cc:194] Registered new tserver with Master: 58fd9c946b964edc8a350361334122d5 (127.4.217.2:34291)
I20260808 04:45:13.776504 8897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:56841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:45:13.794059 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46793
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:13.913086 9473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:13.913324 9473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:13.913368 9473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:13.917403 9473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:13.917459 9473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:13.917550 9473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:13.923126 9473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:13.924191 9473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:13.925320 9473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:13.935077 9481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:13.935086 9485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:13.935185 9473 server_base.cc:1034] running on GCE node
W20260808 04:45:13.935308 9487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:13.935806 9473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:13.936450 9473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:13.937636 9473 hybrid_clock.cc:648] HybridClock initialized: now 1786164313937624 us; error 25 us; skew 500 ppm
Aug 08 04:45:13 dist-test-slave-2kcd krb5kdc[8826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:13.940511 9473 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:45:13.941452 9473 webserver.cc:460] Webserver started at http://127.4.217.3:35601/ using document root <none> and password file <none>
I20260808 04:45:13.942013 9473 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:13.942116 9473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:13.942350 9473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:13.944386 9473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "9d07f5c9495b4ffc8a004b95b0c6bc3b"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "9e98740179ceeb3509f2826fa81c7f82"
server_key_iv: "4a0f0abef4401ece1c1a9f3c8623ba42"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.945014 9473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "9d07f5c9495b4ffc8a004b95b0c6bc3b"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "9e98740179ceeb3509f2826fa81c7f82"
server_key_iv: "4a0f0abef4401ece1c1a9f3c8623ba42"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.948639 9473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:45:13.951538 9498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:13.952841 9473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:13.952986 9473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "9d07f5c9495b4ffc8a004b95b0c6bc3b"
format_stamp: "Formatted at 2026-08-08 04:45:13 on dist-test-slave-2kcd"
server_key: "9e98740179ceeb3509f2826fa81c7f82"
server_key_iv: "4a0f0abef4401ece1c1a9f3c8623ba42"
server_key_version: "encryptionkey@0"
I20260808 04:45:13.953104 9473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:14.008087 9473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:14.011606 9473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:14.011827 9473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:14.012476 9473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:14.013377 9473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:14.013453 9473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:14.013531 9473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:14.013578 9473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:14.022886 9473 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:34463
I20260808 04:45:14.022922 9688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:34463 every 8 connection(s)
I20260808 04:45:14.023799 9473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:14.032430 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9473
I20260808 04:45:14.032536 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:14.032914 4964 external_mini_cluster.cc:1428] Setting key b4b25e2b53e4c11f23d8a845823655a8
Aug 08 04:45:14 dist-test-slave-2kcd krb5kdc[8826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:14.038198 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:14.025696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:43567 (local address 127.4.217.62:46793)
0808 04:45:14.025866 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:14.025871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:14.026707 (+ 836us) server_negotiation.cc:407] Connection header received
0808 04:45:14.028096 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:14.028099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:14.028149 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:14.028224 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:14.029471 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:14.029999 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:14.031122 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:14.031327 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:14.031404 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:14.034460 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:14.034464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:14.034467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:14.034497 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:14.036270 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:14.036747 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:14.036750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:14.036751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:14.036917 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:14.037229 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:14.037231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:14.037232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:14.037413 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:14.037489 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:14.037982 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 04:45:14.038080 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:45:14.039008 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:14.026086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:46793 (local address 127.4.217.3:43567)
0808 04:45:14.026550 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:14.026595 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:14.027885 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:14.028407 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:14.028414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:14.028772 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:14.029329 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:14.029342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:14.030122 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:14.030125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:14.031008 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:14.031018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:14.031473 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:14.031476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:14.031615 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:14.032263 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:14.032280 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:14.034267 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:14.036383 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:14.036388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:14.036399 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:14.036656 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:14.037008 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:14.037011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:14.037012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:14.037145 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:14.037505 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:14.037510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:14.037722 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 04:45:14.037899 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 04:45:14.038149 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260808 04:45:14.040107 9689 heartbeater.cc:347] Connected to a master server at 127.4.217.62:46793
I20260808 04:45:14.040354 9689 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:14.040915 9689 heartbeater.cc:511] Master 127.4.217.62:46793 requested a full tablet report, sending...
I20260808 04:45:14.041968 8897 ts_manager.cc:194] Registered new tserver with Master: 9d07f5c9495b4ffc8a004b95b0c6bc3b (127.4.217.3:34463)
I20260808 04:45:14.042544 8897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:43567
I20260808 04:45:14.047300 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:45:14.063050 8977 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:14.061633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.4.217.62:46793)
0808 04:45:14.061756 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:14.061760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:14.061954 (+ 194us) server_negotiation.cc:407] Connection header received
0808 04:45:14.062055 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:14.062057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:14.062096 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:14.062166 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:14.062878 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35272: BlockingRecv error: recv got EOF from 127.0.0.1:35272 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:45:14.064720 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9010
I20260808 04:45:14.084362 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9230
I20260808 04:45:14.096230 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9473
I20260808 04:45:14.119184 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 8845
2026-08-08T04:45:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2464 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:14 dist-test-slave-2kcd krb5kdc[9705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:14 dist-test-slave-2kcd krb5kdc[9705](info): set up 2 sockets
Aug 08 04:45:14 dist-test-slave-2kcd krb5kdc[9705](info): commencing operation
krb5kdc: starting...
W20260808 04:45:15.377509 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:15 dist-test-slave-2kcd krb5kdc[9705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:15.505482 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:39979
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:39979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:15.616753 9730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:15.617066 9730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:15.617149 9730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:15.621137 9730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:15.621220 9730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:15.621256 9730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:15.621311 9730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:15.621351 9730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:15.626534 9730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:39979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:39979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:15.627534 9730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:15.628582 9730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:15.637084 9740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:15.637181 9730 server_base.cc:1034] running on GCE node
W20260808 04:45:15.637228 9741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:15.637105 9746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:15.637912 9730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:15.639071 9730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:15.640278 9730 hybrid_clock.cc:648] HybridClock initialized: now 1786164315640259 us; error 31 us; skew 500 ppm
Aug 08 04:45:15 dist-test-slave-2kcd krb5kdc[9705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164315, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:15.643127 9730 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:15.644032 9730 webserver.cc:460] Webserver started at http://127.4.217.62:45697/ using document root <none> and password file <none>
I20260808 04:45:15.644606 9730 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:15.644701 9730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:15.645001 9730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:15.647045 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "c6030bdd605945a994a803070d225d97"
format_stamp: "Formatted at 2026-08-08 04:45:15 on dist-test-slave-2kcd"
server_key: "0d94e836447a6135401c7f6583a077e7"
server_key_iv: "b38c622c23c2029878620ff37c80f568"
server_key_version: "encryptionkey@0"
I20260808 04:45:15.647622 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "c6030bdd605945a994a803070d225d97"
format_stamp: "Formatted at 2026-08-08 04:45:15 on dist-test-slave-2kcd"
server_key: "0d94e836447a6135401c7f6583a077e7"
server_key_iv: "b38c622c23c2029878620ff37c80f568"
server_key_version: "encryptionkey@0"
I20260808 04:45:15.651767 9730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260808 04:45:15.654361 9753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:15.656032 9730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:45:15.656169 9730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "c6030bdd605945a994a803070d225d97"
format_stamp: "Formatted at 2026-08-08 04:45:15 on dist-test-slave-2kcd"
server_key: "0d94e836447a6135401c7f6583a077e7"
server_key_iv: "b38c622c23c2029878620ff37c80f568"
server_key_version: "encryptionkey@0"
I20260808 04:45:15.656291 9730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:15.674739 9730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:15.677639 9730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:15.677798 9730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:15.684760 9730 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:39979
I20260808 04:45:15.685088 9844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:39979 every 8 connection(s)
I20260808 04:45:15.685766 9730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:15.688611 9846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:15.692425 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9730
I20260808 04:45:15.692567 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:15.692889 4964 external_mini_cluster.cc:1428] Setting key 27bec21c6e504b1f6a36554fa98a5dcd
I20260808 04:45:15.694918 9846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97: Bootstrap starting.
I20260808 04:45:15.697659 9846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:15.698544 9846 log.cc:824] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:15.700712 9846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97: No bootstrap required, opened a new log
Aug 08 04:45:15 dist-test-slave-2kcd krb5kdc[9705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:15.704909 9846 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6030bdd605945a994a803070d225d97" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39979 } }
I20260808 04:45:15.705191 9846 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:15.705262 9846 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6030bdd605945a994a803070d225d97, State: Initialized, Role: FOLLOWER
I20260808 04:45:15.705746 9846 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6030bdd605945a994a803070d225d97" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39979 } }
I20260808 04:45:15.705916 9846 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:15.706017 9846 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:15.706123 9846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:15.706965 9846 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6030bdd605945a994a803070d225d97" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39979 } }
I20260808 04:45:15.707350 9846 leader_election.cc:302] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6030bdd605945a994a803070d225d97; no voters:
I20260808 04:45:15.707722 9846 leader_election.cc:288] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:15.707794 9851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:15.708021 9851 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [term 1 LEADER]: Becoming Leader. State: Replica: c6030bdd605945a994a803070d225d97, State: Running, Role: LEADER
I20260808 04:45:15.708344 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:15.694287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.4.217.62:39979)
0808 04:45:15.694723 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:15.694743 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:15.694787 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:45:15.695601 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:15.695626 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:15.696030 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:15.696414 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:15.697197 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:15.698725 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:15.699786 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:15.700427 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:15.700579 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:15.703015 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:15.703023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:15.703050 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:15.703089 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:15.705119 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:15.705668 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:15.705674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:15.705679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:15.705826 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:15.706148 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:15.706152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:15.706154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:15.706596 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:15.706816 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:15.707116 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 04:45:15.707396 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":103,"threads_started":1}
I20260808 04:45:15.708336 9851 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6030bdd605945a994a803070d225d97" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39979 } }
I20260808 04:45:15.709076 9846 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:15.710052 9852 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6030bdd605945a994a803070d225d97. Latest consensus state: current_term: 1 leader_uuid: "c6030bdd605945a994a803070d225d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6030bdd605945a994a803070d225d97" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39979 } } }
I20260808 04:45:15.710193 9852 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:15.710542 9854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6030bdd605945a994a803070d225d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6030bdd605945a994a803070d225d97" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39979 } } }
I20260808 04:45:15.710641 9854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:15.710608 9865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:15.711520 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:45:15.714471 9865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:15.720263 9865 catalog_manager.cc:1347] Generated new cluster ID: 06dacdcf743542fd914d36720d37e3b8
I20260808 04:45:15.720338 9865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:15.730857 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:15.725881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.4.217.62:39979)
0808 04:45:15.726031 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:15.726036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:15.726422 (+ 386us) server_negotiation.cc:407] Connection header received
0808 04:45:15.726589 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:15.726593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:15.726646 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:15.726732 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:15.727251 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:15.727798 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:15.728665 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:15.728980 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:15.729104 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:15.729958 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:15.729962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:15.729965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:15.730409 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:15.730438 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:15.730454 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:15.730564 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:15.730651 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 04:45:15.730709 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:45:15.744547 9865 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:15.745585 9865 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:15.761708 9865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6030bdd605945a994a803070d225d97: Generated new TSK 0
I20260808 04:45:15.762565 9865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:45:15.777806 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9730
2026-08-08T04:45:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1655 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:15 dist-test-slave-2kcd krb5kdc[9899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:15 dist-test-slave-2kcd krb5kdc[9899](info): set up 2 sockets
Aug 08 04:45:15 dist-test-slave-2kcd krb5kdc[9899](info): commencing operation
krb5kdc: starting...
W20260808 04:45:17.033048 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:17 dist-test-slave-2kcd krb5kdc[9899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:17.160086 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:32891
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:17.286126 9921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:17.286723 9921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:17.286841 9921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:17.292577 9921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:17.292665 9921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:17.292706 9921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:17.292742 9921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:17.292793 9921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:17.298269 9921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:17.299383 9921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:17.300438 9921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:17.309021 9930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:17.309130 9921 server_base.cc:1034] running on GCE node
W20260808 04:45:17.309029 9941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:17.309002 9933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:17.309875 9921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:17.311008 9921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:17.312177 9921 hybrid_clock.cc:648] HybridClock initialized: now 1786164317312163 us; error 25 us; skew 500 ppm
Aug 08 04:45:17 dist-test-slave-2kcd krb5kdc[9899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:17.314973 9921 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:17.315856 9921 webserver.cc:460] Webserver started at http://127.4.217.62:43453/ using document root <none> and password file <none>
I20260808 04:45:17.316423 9921 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:17.316524 9921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:17.316759 9921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:17.318876 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "56006558234e4e87847cdb70ad4e5af6"
format_stamp: "Formatted at 2026-08-08 04:45:17 on dist-test-slave-2kcd"
server_key: "6fa6cb9119bf1a99943afb2a1a0f3c66"
server_key_iv: "f11e7238a66de056a3536ea15ba03a30"
server_key_version: "encryptionkey@0"
I20260808 04:45:17.319446 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "56006558234e4e87847cdb70ad4e5af6"
format_stamp: "Formatted at 2026-08-08 04:45:17 on dist-test-slave-2kcd"
server_key: "6fa6cb9119bf1a99943afb2a1a0f3c66"
server_key_iv: "f11e7238a66de056a3536ea15ba03a30"
server_key_version: "encryptionkey@0"
I20260808 04:45:17.323200 9921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:45:17.325609 9954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:17.326854 9921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:17.327021 9921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "56006558234e4e87847cdb70ad4e5af6"
format_stamp: "Formatted at 2026-08-08 04:45:17 on dist-test-slave-2kcd"
server_key: "6fa6cb9119bf1a99943afb2a1a0f3c66"
server_key_iv: "f11e7238a66de056a3536ea15ba03a30"
server_key_version: "encryptionkey@0"
I20260808 04:45:17.327137 9921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:17.342733 9921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:17.345615 9921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:17.345820 9921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:17.353142 9921 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:32891
I20260808 04:45:17.353796 10040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:32891 every 8 connection(s)
I20260808 04:45:17.354194 9921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:17.357340 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:17.357888 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9921
I20260808 04:45:17.358042 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:17.358348 4964 external_mini_cluster.cc:1428] Setting key 458ce1bb339530b3be10d1003025164c
I20260808 04:45:17.363797 10041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6: Bootstrap starting.
I20260808 04:45:17.366580 10041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6: Neither blocks nor log segments found. Creating new log.
Aug 08 04:45:17 dist-test-slave-2kcd krb5kdc[9899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:17.367368 10041 log.cc:824] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:17.369454 10041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6: No bootstrap required, opened a new log
I20260808 04:45:17.372357 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:17.359694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.4.217.62:32891)
0808 04:45:17.360110 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:17.360133 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:17.360170 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:45:17.360874 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:17.360894 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:17.361222 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:17.361495 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:17.362461 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.363414 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.364542 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.365055 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.365183 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:17.367899 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:17.367905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:17.367918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:17.367946 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:17.369624 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:17.370081 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:17.370090 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:17.370100 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:17.370210 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:17.370503 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:17.370510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:17.370512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:17.370889 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:17.371096 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:17.371408 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 04:45:17.371683 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":138,"threads_started":1}
I20260808 04:45:17.373037 10041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56006558234e4e87847cdb70ad4e5af6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 32891 } }
I20260808 04:45:17.373268 10041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:17.373360 10041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56006558234e4e87847cdb70ad4e5af6, State: Initialized, Role: FOLLOWER
I20260808 04:45:17.373842 10041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56006558234e4e87847cdb70ad4e5af6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 32891 } }
I20260808 04:45:17.374001 10041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:17.374082 10041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:17.374174 10041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:17.375151 10041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56006558234e4e87847cdb70ad4e5af6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 32891 } }
I20260808 04:45:17.375432 10041 leader_election.cc:302] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56006558234e4e87847cdb70ad4e5af6; no voters:
I20260808 04:45:17.375662 10041 leader_election.cc:288] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:17.375783 10055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:17.376030 10055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [term 1 LEADER]: Becoming Leader. State: Replica: 56006558234e4e87847cdb70ad4e5af6, State: Running, Role: LEADER
I20260808 04:45:17.376328 10055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56006558234e4e87847cdb70ad4e5af6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 32891 } }
I20260808 04:45:17.376888 10041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:17.377987 10056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56006558234e4e87847cdb70ad4e5af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56006558234e4e87847cdb70ad4e5af6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 32891 } } }
I20260808 04:45:17.378140 10056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:17.378485 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56006558234e4e87847cdb70ad4e5af6. Latest consensus state: current_term: 1 leader_uuid: "56006558234e4e87847cdb70ad4e5af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56006558234e4e87847cdb70ad4e5af6" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 32891 } } }
I20260808 04:45:17.378582 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:17.378659 10077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:17.381726 10077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:17.387440 10077 catalog_manager.cc:1347] Generated new cluster ID: 611caf23d1a14c0f85ac2a3368f9f897
I20260808 04:45:17.387513 10077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:17.397555 10077 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:17.398408 10077 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:17.407370 10077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56006558234e4e87847cdb70ad4e5af6: Generated new TSK 0
I20260808 04:45:17.408116 10077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:45:17.444128 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:32891
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:17.559594 10096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:17.559828 10096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:17.559868 10096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:17.563897 10096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:17.563952 10096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:17.564057 10096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:17.569144 10096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:17.570230 10096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:17.571298 10096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:17.581000 10106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:17.580968 10104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:17.580983 10102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:17.581143 10096 server_base.cc:1034] running on GCE node
I20260808 04:45:17.581528 10096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:17.582127 10096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:17.583288 10096 hybrid_clock.cc:648] HybridClock initialized: now 1786164317583275 us; error 23 us; skew 500 ppm
Aug 08 04:45:17 dist-test-slave-2kcd krb5kdc[9899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:17.586010 10096 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:45:17.586851 10096 webserver.cc:460] Webserver started at http://127.4.217.1:34243/ using document root <none> and password file <none>
I20260808 04:45:17.587397 10096 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:17.587491 10096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:17.587720 10096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:17.589766 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "6b08191781514420a70bb0246d2aecec"
format_stamp: "Formatted at 2026-08-08 04:45:17 on dist-test-slave-2kcd"
server_key: "f64f5915d63c2daf95acf55d179d0def"
server_key_iv: "d560b1d5288d023ec870721a11ebddc1"
server_key_version: "encryptionkey@0"
I20260808 04:45:17.590340 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "6b08191781514420a70bb0246d2aecec"
format_stamp: "Formatted at 2026-08-08 04:45:17 on dist-test-slave-2kcd"
server_key: "f64f5915d63c2daf95acf55d179d0def"
server_key_iv: "d560b1d5288d023ec870721a11ebddc1"
server_key_version: "encryptionkey@0"
I20260808 04:45:17.593961 10096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:17.596311 10119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:17.597528 10096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:17.597666 10096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "6b08191781514420a70bb0246d2aecec"
format_stamp: "Formatted at 2026-08-08 04:45:17 on dist-test-slave-2kcd"
server_key: "f64f5915d63c2daf95acf55d179d0def"
server_key_iv: "d560b1d5288d023ec870721a11ebddc1"
server_key_version: "encryptionkey@0"
I20260808 04:45:17.597776 10096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:17.616896 10096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:17.619798 10096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:17.620023 10096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:17.620643 10096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:17.621631 10096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:17.621706 10096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:17.621773 10096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:17.621817 10096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:17.631038 10096 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:46715
I20260808 04:45:17.631064 10302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:46715 every 8 connection(s)
I20260808 04:45:17.631984 10096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:17.641593 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 10096
I20260808 04:45:17.641741 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
Aug 08 04:45:17 dist-test-slave-2kcd krb5kdc[9899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:17.642099 4964 external_mini_cluster.cc:1428] Setting key dc65733ffc160785bf86df773db727c5
I20260808 04:45:17.646620 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:17.633869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:51833 (local address 127.4.217.62:32891)
0808 04:45:17.634014 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:17.634017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:17.634796 (+ 779us) server_negotiation.cc:407] Connection header received
0808 04:45:17.636220 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:17.636224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:17.636286 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:17.636395 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:17.637660 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.638202 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.639339 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.639578 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.639695 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:17.642627 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:17.642631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:17.642634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:17.642664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:17.644179 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:17.644852 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:17.644856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:17.644857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:17.644951 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:17.645350 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:17.645352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:17.645353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:17.645531 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:17.645609 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:17.646256 (+ 647us) server_negotiation.cc:299] Negotiation successful
0808 04:45:17.646404 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:45:17.647289 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:17.634180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:32891 (local address 127.4.217.1:51833)
0808 04:45:17.634643 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:17.634685 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:17.636004 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:17.636587 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:17.636594 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:17.637023 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:17.637506 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:17.637520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.638336 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.638339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:17.639204 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:17.639214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.639738 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.639742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:17.639921 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:17.640586 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:17.640602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:17.642442 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:17.644305 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:17.644314 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:17.644330 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:17.644704 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:17.645048 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:17.645054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:17.645059 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:17.645251 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:17.645638 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:17.645647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:17.645985 (+ 338us) client_negotiation.cc:769] Sending connection context
0808 04:45:17.646168 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 04:45:17.646414 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260808 04:45:17.648515 10304 heartbeater.cc:347] Connected to a master server at 127.4.217.62:32891
I20260808 04:45:17.648816 10304 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:17.649324 10304 heartbeater.cc:511] Master 127.4.217.62:32891 requested a full tablet report, sending...
I20260808 04:45:17.650962 9979 ts_manager.cc:194] Registered new tserver with Master: 6b08191781514420a70bb0246d2aecec (127.4.217.1:46715)
I20260808 04:45:17.652094 9979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:51833
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:45:17.671852 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:32891
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:17.787791 10310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:17.788100 10310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:17.788146 10310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:17.792172 10310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:17.792225 10310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:17.792281 10310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:17.797432 10310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:17.798446 10310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:17.799403 10310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:17.809157 10321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:17.809171 10318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:17.809314 10310 server_base.cc:1034] running on GCE node
W20260808 04:45:17.809157 10324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:17.809864 10310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:17.810456 10310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:17.811621 10310 hybrid_clock.cc:648] HybridClock initialized: now 1786164317811604 us; error 29 us; skew 500 ppm
Aug 08 04:45:17 dist-test-slave-2kcd krb5kdc[9899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:17.814332 10310 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:45:17.815263 10310 webserver.cc:460] Webserver started at http://127.4.217.2:36935/ using document root <none> and password file <none>
I20260808 04:45:17.815867 10310 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:17.815932 10310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:17.816175 10310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:17.818364 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "2bc8b03931844e928a3d0298c2c96602"
format_stamp: "Formatted at 2026-08-08 04:45:17 on dist-test-slave-2kcd"
server_key: "42fff10bb8ddded7167e2993c9216463"
server_key_iv: "dbce7bdd1fb083e523b83017314ac568"
server_key_version: "encryptionkey@0"
I20260808 04:45:17.818959 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "2bc8b03931844e928a3d0298c2c96602"
format_stamp: "Formatted at 2026-08-08 04:45:17 on dist-test-slave-2kcd"
server_key: "42fff10bb8ddded7167e2993c9216463"
server_key_iv: "dbce7bdd1fb083e523b83017314ac568"
server_key_version: "encryptionkey@0"
I20260808 04:45:17.822866 10310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:45:17.825371 10337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:17.826678 10310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:17.826792 10310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "2bc8b03931844e928a3d0298c2c96602"
format_stamp: "Formatted at 2026-08-08 04:45:17 on dist-test-slave-2kcd"
server_key: "42fff10bb8ddded7167e2993c9216463"
server_key_iv: "dbce7bdd1fb083e523b83017314ac568"
server_key_version: "encryptionkey@0"
I20260808 04:45:17.826881 10310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:17.850380 10310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:17.853368 10310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:17.853536 10310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:17.854102 10310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:17.854928 10310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:17.854975 10310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:17.855023 10310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:17.855047 10310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:17.864347 10310 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:39647
I20260808 04:45:17.864377 10537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:39647 every 8 connection(s)
I20260808 04:45:17.865325 10310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:17.869220 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 10310
I20260808 04:45:17.869349 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:17.869650 4964 external_mini_cluster.cc:1428] Setting key 68d5db2192f7f4fd3c5403b9e30b4e49
Aug 08 04:45:17 dist-test-slave-2kcd krb5kdc[9899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:45:17.880419 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:17.867209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:34431 (local address 127.4.217.62:32891)
0808 04:45:17.867369 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:17.867374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:17.868254 (+ 880us) server_negotiation.cc:407] Connection header received
0808 04:45:17.869879 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:17.869883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:17.869952 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:17.870060 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:17.871429 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.871954 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.873253 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.873655 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.873742 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:17.876579 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:17.876582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:17.876584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:17.876608 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:17.878310 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:17.878842 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:17.878847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:17.878849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:17.878932 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:17.879325 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:17.879328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:17.879330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:17.879515 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:17.879628 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:17.880131 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 04:45:17.880252 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:45:17.881294 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:17.867622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:32891 (local address 127.4.217.2:34431)
0808 04:45:17.868080 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:17.868124 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:17.869627 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:17.870243 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:17.870250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:17.870742 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:17.871280 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:17.871294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.872106 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.872110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:17.873088 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:17.873103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:17.873969 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:17.873975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:17.874201 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:17.874842 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:17.874859 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:17.876404 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:17.878441 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:17.878451 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:17.878467 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:17.878732 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:17.879071 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:17.879074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:17.879077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:17.879222 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:17.879628 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:17.879636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:17.879888 (+ 252us) client_negotiation.cc:769] Sending connection context
0808 04:45:17.880085 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 04:45:17.880409 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260808 04:45:17.882627 10538 heartbeater.cc:347] Connected to a master server at 127.4.217.62:32891
I20260808 04:45:17.882848 10538 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
I20260808 04:45:17.883383 10538 heartbeater.cc:511] Master 127.4.217.62:32891 requested a full tablet report, sending...
I20260808 04:45:17.884465 9979 ts_manager.cc:194] Registered new tserver with Master: 2bc8b03931844e928a3d0298c2c96602 (127.4.217.2:39647)
I20260808 04:45:17.885356 9979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:34431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:45:17.899400 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:32891
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:18.012224 10545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:18.012499 10545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:18.012578 10545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:18.016525 10545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:18.016610 10545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:18.016700 10545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:18.021724 10545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:18.022713 10545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:18.023729 10545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:18.033639 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:18.033669 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:18.033766 10545 server_base.cc:1034] running on GCE node
W20260808 04:45:18.033639 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:18.034415 10545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:18.035075 10545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:18.036248 10545 hybrid_clock.cc:648] HybridClock initialized: now 1786164318036231 us; error 26 us; skew 500 ppm
Aug 08 04:45:18 dist-test-slave-2kcd krb5kdc[9899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164318, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:18.039196 10545 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:45:18.040119 10545 webserver.cc:460] Webserver started at http://127.4.217.3:45779/ using document root <none> and password file <none>
I20260808 04:45:18.040688 10545 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:18.040751 10545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:18.041052 10545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:18.043103 10545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "16bce1e2d466483ab44a70f31d068997"
format_stamp: "Formatted at 2026-08-08 04:45:18 on dist-test-slave-2kcd"
server_key: "93d76c3eb319dec98899ff74e3fb7351"
server_key_iv: "cd0f40d9c481059636067a6564bfd1ad"
server_key_version: "encryptionkey@0"
I20260808 04:45:18.043726 10545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "16bce1e2d466483ab44a70f31d068997"
format_stamp: "Formatted at 2026-08-08 04:45:18 on dist-test-slave-2kcd"
server_key: "93d76c3eb319dec98899ff74e3fb7351"
server_key_iv: "cd0f40d9c481059636067a6564bfd1ad"
server_key_version: "encryptionkey@0"
I20260808 04:45:18.047601 10545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 04:45:18.050206 10569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:18.051527 10545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:45:18.051662 10545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "16bce1e2d466483ab44a70f31d068997"
format_stamp: "Formatted at 2026-08-08 04:45:18 on dist-test-slave-2kcd"
server_key: "93d76c3eb319dec98899ff74e3fb7351"
server_key_iv: "cd0f40d9c481059636067a6564bfd1ad"
server_key_version: "encryptionkey@0"
I20260808 04:45:18.051780 10545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:18.070465 10545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:18.073956 10545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:18.074182 10545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:18.074833 10545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:18.075709 10545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:18.075759 10545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:18.075829 10545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:18.075867 10545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:18.085503 10545 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:39929
I20260808 04:45:18.085541 10745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:39929 every 8 connection(s)
I20260808 04:45:18.086460 10545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:18.096148 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 10545
I20260808 04:45:18.096257 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
Aug 08 04:45:18 dist-test-slave-2kcd krb5kdc[9899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164318, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:18.096665 4964 external_mini_cluster.cc:1428] Setting key b9fd46149933f4e3a2b3d55ec9d1597b
I20260808 04:45:18.100648 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:18.088353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:49869 (local address 127.4.217.62:32891)
0808 04:45:18.088527 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:18.088532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:18.089453 (+ 921us) server_negotiation.cc:407] Connection header received
0808 04:45:18.090885 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:18.090889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:18.090951 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:18.091036 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:18.092211 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:18.092739 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:18.094029 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:18.094256 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:18.094339 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:18.097066 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:18.097070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:18.097075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:18.097106 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:18.098833 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:18.099329 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:18.099332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:18.099334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:18.099409 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:18.099709 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:18.099711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:18.099713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:18.099886 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:18.099964 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:18.100421 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 04:45:18.100528 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:45:18.101608 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:18.088748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:32891 (local address 127.4.217.3:49869)
0808 04:45:18.089294 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:18.089340 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:18.090621 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:18.091206 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:18.091213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:18.091582 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:18.092061 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:18.092075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:18.092903 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:18.092907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:18.093915 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:18.093926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:18.094393 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:18.094396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:18.094555 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:18.095106 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:18.095123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:18.096895 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:18.098955 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:18.098960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:18.098971 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:18.099242 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:18.099494 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:18.099496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:18.099497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:18.099628 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:18.099975 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:18.099981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:18.100216 (+ 235us) client_negotiation.cc:769] Sending connection context
0808 04:45:18.100372 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 04:45:18.100621 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":110,"threads_started":1}
I20260808 04:45:18.102766 10746 heartbeater.cc:347] Connected to a master server at 127.4.217.62:32891
I20260808 04:45:18.103020 10746 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:18.103546 10746 heartbeater.cc:511] Master 127.4.217.62:32891 requested a full tablet report, sending...
I20260808 04:45:18.104594 9979 ts_manager.cc:194] Registered new tserver with Master: 16bce1e2d466483ab44a70f31d068997 (127.4.217.3:39929)
I20260808 04:45:18.105253 9979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:49869
I20260808 04:45:18.110600 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:45:18 dist-test-slave-2kcd krb5kdc[9899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164318, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 04:45:18 dist-test-slave-2kcd krb5kdc[9899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164318, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:18.137681 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:18.128974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.4.217.62:32891)
0808 04:45:18.129174 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:18.129180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:18.129282 (+ 102us) server_negotiation.cc:407] Connection header received
0808 04:45:18.129403 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:18.129406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:18.129455 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:18.129530 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:18.130339 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:18.130932 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:18.132039 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:18.132253 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:18.132338 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:18.134821 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:18.134825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:18.134827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:18.134853 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:18.136098 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:18.136510 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:18.136512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:18.136514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:18.136592 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:18.136885 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:18.136888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:18.136889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:18.137118 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:18.137229 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:18.137443 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 04:45:18.137554 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260808 04:45:18.138118 9979 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42386
I20260808 04:45:18.139213 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 10096
I20260808 04:45:18.156313 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 10310
I20260808 04:45:18.172909 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 10545
I20260808 04:45:18.190995 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9921
2026-08-08T04:45:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2416 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:18 dist-test-slave-2kcd krb5kdc[10773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:18 dist-test-slave-2kcd krb5kdc[10773](info): set up 2 sockets
Aug 08 04:45:18 dist-test-slave-2kcd krb5kdc[10773](info): commencing operation
krb5kdc: starting...
W20260808 04:45:19.474299 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:19 dist-test-slave-2kcd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:19.604235 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33299
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:33299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:19.723651 10797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:19.723922 10797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:19.723960 10797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:19.728003 10797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:19.728063 10797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:19.728082 10797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:19.728125 10797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:19.728161 10797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:19.733502 10797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:33299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:19.734586 10797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:19.735704 10797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:19.744357 10806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:19.744410 10803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:19.744498 10797 server_base.cc:1034] running on GCE node
W20260808 04:45:19.744349 10804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:19.745223 10797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:19.746378 10797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:19.747550 10797 hybrid_clock.cc:648] HybridClock initialized: now 1786164319747535 us; error 24 us; skew 500 ppm
Aug 08 04:45:19 dist-test-slave-2kcd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:19.750530 10797 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:19.751382 10797 webserver.cc:460] Webserver started at http://127.4.217.62:42689/ using document root <none> and password file <none>
I20260808 04:45:19.751953 10797 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:19.752045 10797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:19.752269 10797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:19.754376 10797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "abb6409d48174714996d3af97138b2cb"
format_stamp: "Formatted at 2026-08-08 04:45:19 on dist-test-slave-2kcd"
server_key: "647f5840274ca94fb11e22425c92dc9d"
server_key_iv: "06c4306d4927da2b48701b32fbf7f12f"
server_key_version: "encryptionkey@0"
I20260808 04:45:19.754951 10797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "abb6409d48174714996d3af97138b2cb"
format_stamp: "Formatted at 2026-08-08 04:45:19 on dist-test-slave-2kcd"
server_key: "647f5840274ca94fb11e22425c92dc9d"
server_key_iv: "06c4306d4927da2b48701b32fbf7f12f"
server_key_version: "encryptionkey@0"
I20260808 04:45:19.758682 10797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:45:19.761034 10816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:19.762224 10797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:19.762367 10797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "abb6409d48174714996d3af97138b2cb"
format_stamp: "Formatted at 2026-08-08 04:45:19 on dist-test-slave-2kcd"
server_key: "647f5840274ca94fb11e22425c92dc9d"
server_key_iv: "06c4306d4927da2b48701b32fbf7f12f"
server_key_version: "encryptionkey@0"
I20260808 04:45:19.762475 10797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:19.777160 10797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:19.779886 10797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:19.780090 10797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:19.787072 10797 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:33299
I20260808 04:45:19.787142 10905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:33299 every 8 connection(s)
I20260808 04:45:19.788034 10797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:19.790889 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:19.791221 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 10797
I20260808 04:45:19.791347 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:19.791688 4964 external_mini_cluster.cc:1428] Setting key 4e55726a0d6683659b34086876b8f6b7
I20260808 04:45:19.797564 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb: Bootstrap starting.
I20260808 04:45:19.799913 10906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb: Neither blocks nor log segments found. Creating new log.
Aug 08 04:45:19 dist-test-slave-2kcd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:19.800904 10906 log.cc:824] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:19.803022 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb: No bootstrap required, opened a new log
I20260808 04:45:19.805758 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:19.793160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.4.217.62:33299)
0808 04:45:19.793606 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:19.793621 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:19.793658 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:45:19.794362 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:19.794380 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:19.794697 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:19.795051 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:19.795896 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:19.796951 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:19.797873 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:19.798362 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:19.798547 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:19.801295 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:19.801303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:19.801320 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:19.801356 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:19.803009 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:19.803465 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:19.803470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:19.803475 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:19.803593 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:19.803898 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:19.803902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:19.803903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:19.804277 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:19.804488 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:19.804740 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 04:45:19.805064 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":138,"threads_started":1}
I20260808 04:45:19.806871 10906 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb6409d48174714996d3af97138b2cb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33299 } }
I20260808 04:45:19.807128 10906 raft_consensus.cc:374] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:19.807209 10906 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb6409d48174714996d3af97138b2cb, State: Initialized, Role: FOLLOWER
I20260808 04:45:19.807675 10906 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb6409d48174714996d3af97138b2cb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33299 } }
I20260808 04:45:19.807832 10906 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:19.807930 10906 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:19.808053 10906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:19.808959 10906 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb6409d48174714996d3af97138b2cb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33299 } }
I20260808 04:45:19.809300 10906 leader_election.cc:302] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb6409d48174714996d3af97138b2cb; no voters:
I20260808 04:45:19.809610 10906 leader_election.cc:288] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:19.809768 10916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:19.810010 10916 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [term 1 LEADER]: Becoming Leader. State: Replica: abb6409d48174714996d3af97138b2cb, State: Running, Role: LEADER
I20260808 04:45:19.810395 10916 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb6409d48174714996d3af97138b2cb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33299 } }
I20260808 04:45:19.810823 10906 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:19.812047 10917 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abb6409d48174714996d3af97138b2cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb6409d48174714996d3af97138b2cb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33299 } } }
I20260808 04:45:19.812186 10917 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:19.812536 10919 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb6409d48174714996d3af97138b2cb. Latest consensus state: current_term: 1 leader_uuid: "abb6409d48174714996d3af97138b2cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb6409d48174714996d3af97138b2cb" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33299 } } }
I20260808 04:45:19.812662 10919 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb [sys.catalog]: This master's current role is: LEADER
W20260808 04:45:19.815677 10936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:45:19.815769 10936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:45:19.815845 10928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:19.816540 10928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:19.822194 10928 catalog_manager.cc:1347] Generated new cluster ID: 7a6f22331cc24a9988cd2759e53eb05c
I20260808 04:45:19.822314 10928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:19.844554 10928 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:19.845816 10928 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:19.859632 10928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abb6409d48174714996d3af97138b2cb: Generated new TSK 0
I20260808 04:45:19.860451 10928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:45:19.892119 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33299
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:20.007781 10948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:20.008090 10948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:20.008162 10948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:20.012434 10948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:20.012516 10948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:20.012609 10948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:20.018060 10948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:20.019207 10948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:20.020351 10948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:20.030480 10958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:20.030573 10948 server_base.cc:1034] running on GCE node
W20260808 04:45:20.030607 10955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:20.030673 10956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:20.031174 10948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:20.031790 10948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:20.032992 10948 hybrid_clock.cc:648] HybridClock initialized: now 1786164320032974 us; error 51 us; skew 500 ppm
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:20.035863 10948 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:45:20.036734 10948 webserver.cc:460] Webserver started at http://127.4.217.1:45667/ using document root <none> and password file <none>
I20260808 04:45:20.037334 10948 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:20.037437 10948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:20.037678 10948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:20.039753 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "029e90e91b084925b71bcedeed12a636"
format_stamp: "Formatted at 2026-08-08 04:45:20 on dist-test-slave-2kcd"
server_key: "013d5314d8bc2190cb3c68bd69daa2b9"
server_key_iv: "808cf57114a70f48a9f4ef664330d639"
server_key_version: "encryptionkey@0"
I20260808 04:45:20.040323 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "029e90e91b084925b71bcedeed12a636"
format_stamp: "Formatted at 2026-08-08 04:45:20 on dist-test-slave-2kcd"
server_key: "013d5314d8bc2190cb3c68bd69daa2b9"
server_key_iv: "808cf57114a70f48a9f4ef664330d639"
server_key_version: "encryptionkey@0"
I20260808 04:45:20.043988 10948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:45:20.046419 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:20.047580 10948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:45:20.047720 10948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "029e90e91b084925b71bcedeed12a636"
format_stamp: "Formatted at 2026-08-08 04:45:20 on dist-test-slave-2kcd"
server_key: "013d5314d8bc2190cb3c68bd69daa2b9"
server_key_iv: "808cf57114a70f48a9f4ef664330d639"
server_key_version: "encryptionkey@0"
I20260808 04:45:20.047832 10948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:20.063834 10948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:20.066926 10948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:20.067131 10948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:20.067744 10948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:20.068655 10948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:20.068732 10948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:20.068846 10948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:20.068917 10948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:20.077868 10948 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:45693
I20260808 04:45:20.077912 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:45693 every 8 connection(s)
I20260808 04:45:20.078789 10948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:20.089113 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 10948
I20260808 04:45:20.089262 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:20.089610 4964 external_mini_cluster.cc:1428] Setting key 2b17793ef2960bbae116429743f08893
I20260808 04:45:20.093358 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:20.080670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:58559 (local address 127.4.217.62:33299)
0808 04:45:20.080871 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:20.080875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:20.081704 (+ 829us) server_negotiation.cc:407] Connection header received
0808 04:45:20.083138 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:20.083142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:20.083205 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:20.083319 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:20.084504 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.085118 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.086285 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.086522 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.086636 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:20.089214 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:20.089219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:20.089225 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:20.089267 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:20.091066 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.091577 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.091581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.091583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.091687 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.092020 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.092026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.092030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.092298 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:20.092409 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:20.092953 (+ 544us) server_negotiation.cc:299] Negotiation successful
0808 04:45:20.093098 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:45:20.094316 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:20.081079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33299 (local address 127.4.217.1:58559)
0808 04:45:20.081544 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:20.081587 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:20.082904 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:20.083508 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:20.083515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:20.083872 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:20.084344 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:20.084359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.085252 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.085262 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:20.086149 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:20.086159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.086680 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.086684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:20.086863 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:20.087441 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:20.087458 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:20.089004 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:20.091191 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:20.091196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:20.091209 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:20.091464 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:20.091789 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:20.091791 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:20.091792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:20.091923 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:20.092404 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:20.092409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:20.092654 (+ 245us) client_negotiation.cc:769] Sending connection context
0808 04:45:20.092895 (+ 241us) client_negotiation.cc:240] Negotiation successful
0808 04:45:20.093179 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260808 04:45:20.095602 11133 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33299
I20260808 04:45:20.095862 11133 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:20.096364 11133 heartbeater.cc:511] Master 127.4.217.62:33299 requested a full tablet report, sending...
I20260808 04:45:20.097975 10848 ts_manager.cc:194] Registered new tserver with Master: 029e90e91b084925b71bcedeed12a636 (127.4.217.1:45693)
I20260808 04:45:20.099156 10848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:58559
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:45:20.120255 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33299
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:20.240859 11142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:20.241092 11142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:20.241168 11142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:20.245242 11142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:20.245292 11142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:20.245393 11142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:20.252164 11142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:20.253528 11142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:20.254946 11142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:20.265316 11148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:20.265352 11153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:20.265347 11158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:20.265760 11142 server_base.cc:1034] running on GCE node
I20260808 04:45:20.266146 11142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:20.266714 11142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:20.267876 11142 hybrid_clock.cc:648] HybridClock initialized: now 1786164320267859 us; error 39 us; skew 500 ppm
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:20.270701 11142 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:45:20.271551 11142 webserver.cc:460] Webserver started at http://127.4.217.2:35093/ using document root <none> and password file <none>
I20260808 04:45:20.272101 11142 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:20.272163 11142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:20.272414 11142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:20.274418 11142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "90e1b12786da48428b9b1e6f23f72dad"
format_stamp: "Formatted at 2026-08-08 04:45:20 on dist-test-slave-2kcd"
server_key: "0d144bb74d8bb8171b734450da0558ee"
server_key_iv: "cd2021750c9c76684f744d9a7f243f5a"
server_key_version: "encryptionkey@0"
I20260808 04:45:20.275002 11142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "90e1b12786da48428b9b1e6f23f72dad"
format_stamp: "Formatted at 2026-08-08 04:45:20 on dist-test-slave-2kcd"
server_key: "0d144bb74d8bb8171b734450da0558ee"
server_key_iv: "cd2021750c9c76684f744d9a7f243f5a"
server_key_version: "encryptionkey@0"
I20260808 04:45:20.278693 11142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:45:20.281067 11170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:20.282228 11142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:20.282367 11142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "90e1b12786da48428b9b1e6f23f72dad"
format_stamp: "Formatted at 2026-08-08 04:45:20 on dist-test-slave-2kcd"
server_key: "0d144bb74d8bb8171b734450da0558ee"
server_key_iv: "cd2021750c9c76684f744d9a7f243f5a"
server_key_version: "encryptionkey@0"
I20260808 04:45:20.282495 11142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:20.304211 11142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:20.307399 11142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:20.307607 11142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:20.308178 11142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:20.309065 11142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:20.309136 11142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:20.309211 11142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:20.309263 11142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:20.320300 11142 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:34127
I20260808 04:45:20.320355 11357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:34127 every 8 connection(s)
I20260808 04:45:20.321238 11142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:20.327873 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 11142
I20260808 04:45:20.327970 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:20.328299 4964 external_mini_cluster.cc:1428] Setting key 273e619d67a1923d31596e7af02f72c4
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:20.335316 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:20.322972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:55387 (local address 127.4.217.62:33299)
0808 04:45:20.323112 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:20.323115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:20.323873 (+ 758us) server_negotiation.cc:407] Connection header received
0808 04:45:20.325276 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:20.325279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:20.325328 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:20.325403 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:20.326528 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.327056 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.328281 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.328611 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.328727 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:20.331372 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:20.331379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:20.331385 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:20.331422 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:20.332978 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.333611 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.333614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.333616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.333692 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.334090 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.334092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.334094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.334272 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:20.334348 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:20.335024 (+ 676us) server_negotiation.cc:299] Negotiation successful
0808 04:45:20.335191 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:20.336591 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:20.323270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33299 (local address 127.4.217.2:55387)
0808 04:45:20.323708 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:20.323749 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:20.325077 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:20.325574 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:20.325581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:20.325926 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:20.326395 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:20.326407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.327166 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.327169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:20.328147 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:20.328157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.328932 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.328936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:20.329080 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:20.329769 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:20.329785 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:20.331203 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:20.333112 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:20.333120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:20.333136 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:20.333505 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:20.333783 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:20.333788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:20.333793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:20.333983 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:20.334380 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:20.334389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:20.334733 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 04:45:20.334989 (+ 256us) client_negotiation.cc:240] Negotiation successful
0808 04:45:20.335381 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260808 04:45:20.337934 11358 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33299
I20260808 04:45:20.338200 11358 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:20.338704 11358 heartbeater.cc:511] Master 127.4.217.62:33299 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:45:20.339852 10848 ts_manager.cc:194] Registered new tserver with Master: 90e1b12786da48428b9b1e6f23f72dad (127.4.217.2:34127)
I20260808 04:45:20.340430 10848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:55387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:45:20.359622 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33299
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:20.474934 11369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:20.475221 11369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:20.475291 11369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:20.479372 11369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:20.479458 11369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:20.479555 11369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:20.484736 11369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:20.485906 11369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:20.487037 11369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:20.496824 11380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:20.496817 11379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:20.496775 11383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:20.497125 11369 server_base.cc:1034] running on GCE node
I20260808 04:45:20.497539 11369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:20.498179 11369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:20.499375 11369 hybrid_clock.cc:648] HybridClock initialized: now 1786164320499359 us; error 26 us; skew 500 ppm
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:20.502146 11369 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:45:20.502977 11369 webserver.cc:460] Webserver started at http://127.4.217.3:38123/ using document root <none> and password file <none>
I20260808 04:45:20.503549 11369 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:20.503643 11369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:20.503876 11369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:20.506007 11369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "b63d512e8f35447e98ba3482d1cafd23"
format_stamp: "Formatted at 2026-08-08 04:45:20 on dist-test-slave-2kcd"
server_key: "2e43b33760abe263ecb0e31e4147686b"
server_key_iv: "831bb01f95701e3c82a2780288b72b22"
server_key_version: "encryptionkey@0"
I20260808 04:45:20.506629 11369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "b63d512e8f35447e98ba3482d1cafd23"
format_stamp: "Formatted at 2026-08-08 04:45:20 on dist-test-slave-2kcd"
server_key: "2e43b33760abe263ecb0e31e4147686b"
server_key_iv: "831bb01f95701e3c82a2780288b72b22"
server_key_version: "encryptionkey@0"
I20260808 04:45:20.510277 11369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:45:20.512976 11392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:20.514163 11369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:20.514302 11369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "b63d512e8f35447e98ba3482d1cafd23"
format_stamp: "Formatted at 2026-08-08 04:45:20 on dist-test-slave-2kcd"
server_key: "2e43b33760abe263ecb0e31e4147686b"
server_key_iv: "831bb01f95701e3c82a2780288b72b22"
server_key_version: "encryptionkey@0"
I20260808 04:45:20.514415 11369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:20.535159 11369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:20.538172 11369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:20.538380 11369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:20.538992 11369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:20.539839 11369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:20.539923 11369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:20.539999 11369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:20.540035 11369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:20.548825 11369 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:45453
I20260808 04:45:20.548841 11574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:45453 every 8 connection(s)
I20260808 04:45:20.549757 11369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:20.556448 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 11369
I20260808 04:45:20.556608 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:20.556963 4964 external_mini_cluster.cc:1428] Setting key 0469991d4a81c849c69ac9346b6d4241
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:20.564100 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:20.551638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:45887 (local address 127.4.217.62:33299)
0808 04:45:20.551781 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:20.551785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:20.552715 (+ 930us) server_negotiation.cc:407] Connection header received
0808 04:45:20.554240 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:20.554243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:20.554307 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:20.554391 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:20.555544 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.556086 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.557458 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.557740 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.557851 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:20.560720 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:20.560724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:20.560726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:20.560752 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:20.562272 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.562743 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.562746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.562747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.562822 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.563125 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.563127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.563129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.563302 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:20.563377 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:20.563869 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 04:45:20.563973 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:45:20.565039 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:20.552014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33299 (local address 127.4.217.3:45887)
0808 04:45:20.552570 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:20.552612 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:20.554009 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:20.554560 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:20.554567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:20.554912 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:20.555386 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:20.555400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.556220 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.556226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:20.557328 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:20.557338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.557923 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.557928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:20.558151 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:20.558813 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:20.558831 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:20.560564 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:20.562376 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:20.562380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:20.562393 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:20.562656 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:20.562904 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:20.562906 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:20.562907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:20.563043 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:20.563392 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:20.563397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:20.563660 (+ 263us) client_negotiation.cc:769] Sending connection context
0808 04:45:20.563819 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 04:45:20.564076 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":126,"threads_started":1}
I20260808 04:45:20.566316 11576 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33299
I20260808 04:45:20.566617 11576 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:20.567150 11576 heartbeater.cc:511] Master 127.4.217.62:33299 requested a full tablet report, sending...
I20260808 04:45:20.568208 10848 ts_manager.cc:194] Registered new tserver with Master: b63d512e8f35447e98ba3482d1cafd23 (127.4.217.3:45453)
I20260808 04:45:20.568764 10848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:45887
I20260808 04:45:20.571542 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:45:20.603412 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:20.589720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.4.217.1:45693)
0808 04:45:20.590039 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:20.590056 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:20.590080 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:45:20.590134 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:20.590141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:20.590340 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:20.590467 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:20.591139 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.591932 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.592737 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.593175 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.593259 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:20.600217 (+ 6958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:20.600228 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:20.600243 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:20.600269 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:20.601811 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.602220 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.602225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.602226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.602304 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.602547 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.602549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.602550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.602797 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:20.602965 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:20.603134 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 04:45:20.603265 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:20.614765 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:20.606473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.4.217.62:33299)
0808 04:45:20.606621 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:20.606628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:20.606647 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:45:20.606804 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:20.606808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:20.606864 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:20.606951 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:20.607558 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.608084 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.608944 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:20.609190 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:20.609303 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:20.611619 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:20.611623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:20.611625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:20.611650 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:20.613062 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.613475 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.613479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.613481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.613568 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:20.613848 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:20.613852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:20.613853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:20.614090 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:20.614235 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:20.614473 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 04:45:20.614593 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 04:45:20.615173 10848 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45888
I20260808 04:45:20.615816 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 10948
I20260808 04:45:20.627763 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 11142
I20260808 04:45:20.651746 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 11369
I20260808 04:45:20.665933 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 10797
2026-08-08T04:45:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2474 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[11596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[11596](info): set up 2 sockets
Aug 08 04:45:20 dist-test-slave-2kcd krb5kdc[11596](info): commencing operation
krb5kdc: starting...
W20260808 04:45:21.957267 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:22 dist-test-slave-2kcd krb5kdc[11596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:22.082015 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:45045
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:22.192551 11614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:22.192852 11614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:22.192934 11614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:22.196867 11614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:22.196949 11614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:22.196987 11614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:22.197050 11614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:22.197088 11614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:22.202761 11614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:45045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:22.203828 11614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:22.204921 11614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:22.213038 11627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:22.213176 11614 server_base.cc:1034] running on GCE node
W20260808 04:45:22.213064 11626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:22.213109 11629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:22.213966 11614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:22.215365 11614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:22.216609 11614 hybrid_clock.cc:648] HybridClock initialized: now 1786164322216592 us; error 37 us; skew 500 ppm
Aug 08 04:45:22 dist-test-slave-2kcd krb5kdc[11596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:22.219457 11614 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:22.220223 11614 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:22.220323 11614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:22.220570 11614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:22.222750 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "4ac37b18adeb4b3593200f2691e49f5f"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "f9fd57f4ff19f82a7e80471fed4afa87"
server_key_iv: "084e60eab6345bf0145b61ddc486c167"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.223340 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "4ac37b18adeb4b3593200f2691e49f5f"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "f9fd57f4ff19f82a7e80471fed4afa87"
server_key_iv: "084e60eab6345bf0145b61ddc486c167"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.227181 11614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:45:22.229635 11638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.230825 11614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:22.230973 11614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "4ac37b18adeb4b3593200f2691e49f5f"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "f9fd57f4ff19f82a7e80471fed4afa87"
server_key_iv: "084e60eab6345bf0145b61ddc486c167"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.231104 11614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:22.245936 11614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:22.249042 11614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:22.249253 11614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:22.259083 11614 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:45045
I20260808 04:45:22.259514 11724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:45045 every 8 connection(s)
I20260808 04:45:22.259918 11614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:22.263023 11726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:22.267750 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 11614
I20260808 04:45:22.267884 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:22.268203 4964 external_mini_cluster.cc:1428] Setting key d3d77dded533d20054aa6d35c760d0ad
I20260808 04:45:22.269059 11726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f: Bootstrap starting.
I20260808 04:45:22.271481 11726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:22.272495 11726 log.cc:824] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:22.274674 11726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f: No bootstrap required, opened a new log
I20260808 04:45:22.278502 11726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac37b18adeb4b3593200f2691e49f5f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45045 } }
I20260808 04:45:22.278729 11726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:22.278765 11726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac37b18adeb4b3593200f2691e49f5f, State: Initialized, Role: FOLLOWER
I20260808 04:45:22.279214 11726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac37b18adeb4b3593200f2691e49f5f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45045 } }
I20260808 04:45:22.279340 11726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:22.279388 11726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:22.279462 11726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:22.280329 11726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac37b18adeb4b3593200f2691e49f5f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45045 } }
I20260808 04:45:22.280720 11726 leader_election.cc:302] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ac37b18adeb4b3593200f2691e49f5f; no voters:
I20260808 04:45:22.281165 11726 leader_election.cc:288] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:22.281282 11731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:22.281590 11731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [term 1 LEADER]: Becoming Leader. State: Replica: 4ac37b18adeb4b3593200f2691e49f5f, State: Running, Role: LEADER
I20260808 04:45:22.281963 11731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac37b18adeb4b3593200f2691e49f5f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45045 } }
Aug 08 04:45:22 dist-test-slave-2kcd krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:22.282639 11726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:22.284250 11733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ac37b18adeb4b3593200f2691e49f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac37b18adeb4b3593200f2691e49f5f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45045 } } }
I20260808 04:45:22.284286 11735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ac37b18adeb4b3593200f2691e49f5f. Latest consensus state: current_term: 1 leader_uuid: "4ac37b18adeb4b3593200f2691e49f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac37b18adeb4b3593200f2691e49f5f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45045 } } }
I20260808 04:45:22.284380 11733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:22.284412 11735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:22.284754 11751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:22.288508 11751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:22.289242 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:22.269719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.4.217.62:45045)
0808 04:45:22.270154 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:22.270169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:22.270223 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:45:22.270837 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:22.270853 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:22.271188 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:22.271464 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:22.272405 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.273658 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.274919 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.275413 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.275543 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:22.282877 (+ 7334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:22.282884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:22.282901 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:22.282935 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:22.285722 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:22.286284 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:22.286289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:22.286294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:22.286405 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:22.286763 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:22.286766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:22.286768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:22.287309 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:22.287520 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:22.287843 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 04:45:22.288193 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":138,"threads_started":1}
I20260808 04:45:22.294420 11751 catalog_manager.cc:1347] Generated new cluster ID: a2f7d0ee1de04c9a9ad9b3f7fb4ab05f
I20260808 04:45:22.294548 11751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:22.301945 11751 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:22.303194 11751 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:22.312546 11751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f: Generated new TSK 0
I20260808 04:45:22.313297 11751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:45:22.352058 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45045
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:22.479645 11765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:22.479868 11765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:22.479911 11765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:22.483918 11765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:22.483968 11765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:22.484025 11765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:22.489143 11765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:22.489996 11765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:22.490902 11765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:22.500070 11774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:22.500064 11780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:22.500061 11773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:22.500358 11765 server_base.cc:1034] running on GCE node
I20260808 04:45:22.500968 11765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:22.501607 11765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:22.502803 11765 hybrid_clock.cc:648] HybridClock initialized: now 1786164322502790 us; error 25 us; skew 500 ppm
Aug 08 04:45:22 dist-test-slave-2kcd krb5kdc[11596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:22.505527 11765 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:45:22.506239 11765 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:22.506336 11765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:22.506582 11765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:22.508572 11765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "2961fa5e16df418cbe8951556f5acee4"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "376b21b1df54b70ae62c2096baf8d8d4"
server_key_iv: "d1638361a5ea26bccdfda3068fc7805a"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.509232 11765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "2961fa5e16df418cbe8951556f5acee4"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "376b21b1df54b70ae62c2096baf8d8d4"
server_key_iv: "d1638361a5ea26bccdfda3068fc7805a"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.512852 11765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:45:22.515202 11789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.516359 11765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:22.516505 11765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "2961fa5e16df418cbe8951556f5acee4"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "376b21b1df54b70ae62c2096baf8d8d4"
server_key_iv: "d1638361a5ea26bccdfda3068fc7805a"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.516618 11765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:22.537155 11765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:22.539942 11765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:22.540138 11765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:22.540699 11765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:22.541581 11765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:22.541646 11765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.541709 11765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:22.541762 11765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.550523 11765 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:44073
I20260808 04:45:22.550544 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:44073 every 8 connection(s)
I20260808 04:45:22.551090 11765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
Aug 08 04:45:22 dist-test-slave-2kcd krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:22.560150 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 11765
I20260808 04:45:22.560283 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:22.560560 4964 external_mini_cluster.cc:1428] Setting key 1d410b9bf57e9d20cc060abc90d2f2fe
I20260808 04:45:22.564342 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:22.552821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:56533 (local address 127.4.217.62:45045)
0808 04:45:22.552958 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:22.552962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:22.554111 (+ 1149us) server_negotiation.cc:407] Connection header received
0808 04:45:22.554964 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:22.554968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:22.555030 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:22.555125 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:22.556214 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.556758 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.557893 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.558127 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.558207 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:22.560557 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:22.560560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:22.560562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:22.560587 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:22.562279 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:22.562798 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:22.562803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:22.562804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:22.562896 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:22.563261 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:22.563264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:22.563266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:22.563452 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:22.563559 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:22.564049 (+ 490us) server_negotiation.cc:299] Negotiation successful
0808 04:45:22.564173 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:22.565152 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:22.553163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:45045 (local address 127.4.217.1:56533)
0808 04:45:22.553963 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:22.554002 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:22.554769 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:22.555282 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:22.555289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:22.555615 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:22.556073 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:22.556087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.556929 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.556931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:22.557779 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:22.557788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.558258 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.558261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:22.558396 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:22.558871 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:22.558886 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:22.560393 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:22.562398 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:22.562404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:22.562418 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:22.562683 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:22.562996 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:22.563000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:22.563002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:22.563157 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:22.563582 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:22.563588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:22.563813 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 04:45:22.563996 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 04:45:22.564298 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
I20260808 04:45:22.566370 11960 heartbeater.cc:347] Connected to a master server at 127.4.217.62:45045
I20260808 04:45:22.566619 11960 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:22.567130 11960 heartbeater.cc:511] Master 127.4.217.62:45045 requested a full tablet report, sending...
I20260808 04:45:22.568985 11669 ts_manager.cc:194] Registered new tserver with Master: 2961fa5e16df418cbe8951556f5acee4 (127.4.217.1:44073)
I20260808 04:45:22.570720 11669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:56533
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:45:22.589769 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45045
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:22.704154 11970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:22.704437 11970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:22.704520 11970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:22.708599 11970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:22.708668 11970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:22.708777 11970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:22.714216 11970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:22.715377 11970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:22.716514 11970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:22.726596 11980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:22.726603 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:22.726603 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:22.726657 11970 server_base.cc:1034] running on GCE node
I20260808 04:45:22.727231 11970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:22.727895 11970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:22.729064 11970 hybrid_clock.cc:648] HybridClock initialized: now 1786164322729043 us; error 33 us; skew 500 ppm
Aug 08 04:45:22 dist-test-slave-2kcd krb5kdc[11596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:22.731833 11970 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:45:22.732569 11970 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:22.732667 11970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:22.732963 11970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:22.735220 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "d48ed79f65664922933f0ff9a73eed2a"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "1640f42a1cfb75e4b23a23ac01d359e1"
server_key_iv: "da081149cb21c67d1653ffcd8c60277b"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.735889 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "d48ed79f65664922933f0ff9a73eed2a"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "1640f42a1cfb75e4b23a23ac01d359e1"
server_key_iv: "da081149cb21c67d1653ffcd8c60277b"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.739650 11970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 04:45:22.742115 11994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.743314 11970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:45:22.743453 11970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "d48ed79f65664922933f0ff9a73eed2a"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "1640f42a1cfb75e4b23a23ac01d359e1"
server_key_iv: "da081149cb21c67d1653ffcd8c60277b"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.743577 11970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:22.753535 11970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:22.756263 11970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:22.756455 11970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:22.757077 11970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:22.757899 11970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:22.757972 11970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.758041 11970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:22.758083 11970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.766680 11970 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:38955
I20260808 04:45:22.766748 12169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:38955 every 8 connection(s)
I20260808 04:45:22.767278 11970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:22.775736 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 11970
I20260808 04:45:22.775908 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:22.776225 4964 external_mini_cluster.cc:1428] Setting key 3c6ade0036d15fce981009862bf973cb
Aug 08 04:45:22 dist-test-slave-2kcd krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:22.780507 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:22.769025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:58451 (local address 127.4.217.62:45045)
0808 04:45:22.769169 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:22.769173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:22.770345 (+ 1172us) server_negotiation.cc:407] Connection header received
0808 04:45:22.771165 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:22.771167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:22.771213 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:22.771286 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:22.772387 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.772962 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.774137 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.774338 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.774415 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:22.777068 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:22.777074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:22.777082 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:22.777107 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:22.778501 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:22.779034 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:22.779038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:22.779039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:22.779112 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:22.779458 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:22.779462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:22.779464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:22.779642 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:22.779755 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:22.780223 (+ 468us) server_negotiation.cc:299] Negotiation successful
0808 04:45:22.780338 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:22.781332 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:22.769405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:45045 (local address 127.4.217.2:58451)
0808 04:45:22.770203 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:22.770242 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:22.770979 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:22.771448 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:22.771454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:22.771796 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:22.772255 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:22.772269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.773074 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.773076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:22.774022 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:22.774032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.774466 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.774469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:22.774618 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:22.775141 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:22.775157 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:22.776905 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:22.778633 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:22.778639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:22.778653 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:22.778903 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:22.779210 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:22.779213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:22.779215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:22.779357 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:22.779755 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:22.779761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:22.779989 (+ 228us) client_negotiation.cc:769] Sending connection context
0808 04:45:22.780177 (+ 188us) client_negotiation.cc:240] Negotiation successful
0808 04:45:22.780476 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260808 04:45:22.782491 12170 heartbeater.cc:347] Connected to a master server at 127.4.217.62:45045
I20260808 04:45:22.782752 12170 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:22.783298 12170 heartbeater.cc:511] Master 127.4.217.62:45045 requested a full tablet report, sending...
I20260808 04:45:22.784462 11669 ts_manager.cc:194] Registered new tserver with Master: d48ed79f65664922933f0ff9a73eed2a (127.4.217.2:38955)
I20260808 04:45:22.785143 11669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:58451
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:45:22.804860 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45045
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:22.915733 12178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:22.916025 12178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:22.916088 12178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:22.920056 12178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:22.920113 12178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:22.920188 12178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:22.925284 12178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:22.926175 12178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:22.927081 12178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:22.936442 12186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:22.936458 12183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:22.936627 12178 server_base.cc:1034] running on GCE node
W20260808 04:45:22.936477 12190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:22.937111 12178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:22.937754 12178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:22.938918 12178 hybrid_clock.cc:648] HybridClock initialized: now 1786164322938899 us; error 31 us; skew 500 ppm
Aug 08 04:45:22 dist-test-slave-2kcd krb5kdc[11596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:22.941852 12178 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:45:22.942662 12178 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:22.942727 12178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:22.942888 12178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:22.944754 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "a27ca17f31e942acbabff2b9e6b96d63"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "2731ae4d34b8f6317fe2dbdd3b285a8b"
server_key_iv: "acc68e5bd472f6dccc3b4e1098a924eb"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.945341 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "a27ca17f31e942acbabff2b9e6b96d63"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "2731ae4d34b8f6317fe2dbdd3b285a8b"
server_key_iv: "acc68e5bd472f6dccc3b4e1098a924eb"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.949081 12178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:45:22.951325 12196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.952497 12178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:22.952641 12178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "a27ca17f31e942acbabff2b9e6b96d63"
format_stamp: "Formatted at 2026-08-08 04:45:22 on dist-test-slave-2kcd"
server_key: "2731ae4d34b8f6317fe2dbdd3b285a8b"
server_key_iv: "acc68e5bd472f6dccc3b4e1098a924eb"
server_key_version: "encryptionkey@0"
I20260808 04:45:22.952742 12178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:22.971789 12178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:22.974974 12178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:22.975184 12178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:22.975762 12178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:22.976572 12178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:22.976619 12178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.976687 12178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:22.976729 12178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:22.986275 12178 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:46759
I20260808 04:45:22.986304 12364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:46759 every 8 connection(s)
I20260808 04:45:22.986851 12178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:22.990557 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 12178
I20260808 04:45:22.990702 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:22.991080 4964 external_mini_cluster.cc:1428] Setting key 0d1b84671e92dc1b55c8f1f7110270a1
Aug 08 04:45:22 dist-test-slave-2kcd krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:23.000452 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:22.988486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:56821 (local address 127.4.217.62:45045)
0808 04:45:22.988644 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:22.988647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:22.989787 (+ 1140us) server_negotiation.cc:407] Connection header received
0808 04:45:22.990813 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:22.990816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:22.990881 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:22.990951 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:22.992522 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.993296 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.994451 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.994670 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.994777 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:22.997272 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:22.997275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:22.997278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:22.997302 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:22.998732 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:22.999180 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:22.999183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:22.999184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:22.999258 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:22.999551 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:22.999554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:22.999555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:22.999720 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:22.999795 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.000242 (+ 447us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.000334 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:45:23.001302 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:22.988826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:45045 (local address 127.4.217.3:56821)
0808 04:45:22.989641 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:22.989682 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:22.990544 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:22.991182 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:22.991191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:22.991660 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:22.992329 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:22.992356 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.993426 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.993430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:22.994298 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:22.994308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:22.994830 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:22.994834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:22.995017 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:22.995604 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:22.995621 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:22.997123 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:22.998830 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:22.998835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:22.998845 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:22.999093 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:22.999340 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:22.999342 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:22.999343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:22.999470 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:22.999808 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:22.999812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:23.000036 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 04:45:23.000190 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 04:45:23.000423 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260808 04:45:23.002409 12365 heartbeater.cc:347] Connected to a master server at 127.4.217.62:45045
I20260808 04:45:23.002652 12365 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:23.003101 12365 heartbeater.cc:511] Master 127.4.217.62:45045 requested a full tablet report, sending...
I20260808 04:45:23.004240 11669 ts_manager.cc:194] Registered new tserver with Master: a27ca17f31e942acbabff2b9e6b96d63 (127.4.217.3:46759)
I20260808 04:45:23.005059 11669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:56821
I20260808 04:45:23.005228 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:45:23.014137 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.007271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.4.217.62:45045)
0808 04:45:23.007415 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.007419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.007615 (+ 196us) server_negotiation.cc:407] Connection header received
0808 04:45:23.007697 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.007699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.007737 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.007801 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:23.008455 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.009027 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.009810 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.010017 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.010092 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.011399 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:23.011401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:23.011403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:23.011425 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:23.012624 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:23.013061 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:23.013064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:23.013065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:23.013139 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:23.013369 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:23.013371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:23.013372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:23.013565 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:23.013668 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.013831 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.013933 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:45:23.016995 11669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:23.020025 11669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:23.035135 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.028937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:44073 (local address 127.0.0.1:46856)
0808 04:45:23.029554 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:23.029606 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:23.029736 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:23.030356 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:23.030365 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:23.030398 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:23.030539 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.030550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.032065 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.032069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.033443 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.033454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.034548 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.034552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.034663 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.034703 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 04:45:23.034821 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 04:45:23.034960 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":41,"thread_start_us":97,"threads_started":1}
I20260808 04:45:23.035143 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.029049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.4.217.1:44073)
0808 04:45:23.029805 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.029815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.029841 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:45:23.029923 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.029928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.030166 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.030346 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:23.030695 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.031913 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.033571 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.034229 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.034338 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.034482 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.034860 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.034967 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":130,"threads_started":1}
I20260808 04:45:23.035890 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.030309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:38955 (local address 127.0.0.1:38126)
0808 04:45:23.030654 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:23.030669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:23.030765 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:23.031541 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:23.031543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:23.031557 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:23.031610 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.031613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.033194 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.033197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.034219 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.034227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.035458 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.035461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.035570 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.035599 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:45:23.035683 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:45:23.035760 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"spinlock_wait_cycles":8320,"thread_start_us":90,"threads_started":1}
I20260808 04:45:23.036113 12390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.030477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.4.217.2:38955)
0808 04:45:23.030929 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.030938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.030955 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:45:23.031014 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.031017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.031159 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.031308 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:23.031720 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.033029 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.034334 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.035135 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.035250 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.035386 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.035738 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.035832 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":152,"threads_started":1}
I20260808 04:45:23.036975 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.029793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.4.217.3:46759)
0808 04:45:23.030166 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.030181 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.030875 (+ 694us) server_negotiation.cc:407] Connection header received
0808 04:45:23.030958 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.030964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.031140 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.031315 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:23.031543 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.032600 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.034049 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.034879 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.034976 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.035104 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.036679 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.036807 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":167,"threads_started":1}
I20260808 04:45:23.038681 12057 tablet_service.cc:1467] Processing CreateTablet for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3 (DEFAULT_TABLE table=test-table [id=3ce4c312e2a2479687a574d135dbfdf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:23.038682 11867 tablet_service.cc:1467] Processing CreateTablet for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3 (DEFAULT_TABLE table=test-table [id=3ce4c312e2a2479687a574d135dbfdf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:23.036855 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.028976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:46759 (local address 127.0.0.1:55142)
0808 04:45:23.030644 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:23.030660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:23.030769 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:23.031294 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:23.031297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:23.031317 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:23.031418 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.031427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.032758 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.032764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.033909 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.033921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.035171 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.035174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.036512 (+ 1338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.036547 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:45:23.036629 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:45:23.036695 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1586,"mutex_wait_us":24,"spinlock_wait_cycles":2688,"thread_start_us":88,"threads_started":1}
I20260808 04:45:23.039873 11867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:23.039876 12057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:23.041988 12267 tablet_service.cc:1467] Processing CreateTablet for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3 (DEFAULT_TABLE table=test-table [id=3ce4c312e2a2479687a574d135dbfdf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:23.042888 12267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:23.047859 12391 tablet_bootstrap.cc:492] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4: Bootstrap starting.
I20260808 04:45:23.048163 12396 tablet_bootstrap.cc:492] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: Bootstrap starting.
I20260808 04:45:23.049773 12397 tablet_bootstrap.cc:492] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: Bootstrap starting.
I20260808 04:45:23.051034 12391 tablet_bootstrap.cc:654] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:23.051067 12396 tablet_bootstrap.cc:654] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:23.051903 12397 tablet_bootstrap.cc:654] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:23.052161 12396 log.cc:824] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:23.052241 12391 log.cc:824] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:23.053089 12397 log.cc:824] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:23.054252 12396 tablet_bootstrap.cc:492] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: No bootstrap required, opened a new log
I20260808 04:45:23.054306 12391 tablet_bootstrap.cc:492] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4: No bootstrap required, opened a new log
I20260808 04:45:23.054476 12396 ts_tablet_manager.cc:1397] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:45:23.054502 12391 ts_tablet_manager.cc:1397] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:45:23.055032 12397 tablet_bootstrap.cc:492] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: No bootstrap required, opened a new log
I20260808 04:45:23.055212 12397 ts_tablet_manager.cc:1397] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:45:23.058367 12396 raft_consensus.cc:348] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.058646 12396 raft_consensus.cc:374] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:23.058703 12396 raft_consensus.cc:729] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d48ed79f65664922933f0ff9a73eed2a, State: Initialized, Role: FOLLOWER
I20260808 04:45:23.058568 12391 raft_consensus.cc:348] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.058815 12391 raft_consensus.cc:374] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:23.058879 12391 raft_consensus.cc:729] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961fa5e16df418cbe8951556f5acee4, State: Initialized, Role: FOLLOWER
I20260808 04:45:23.059098 12397 raft_consensus.cc:348] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.059242 12396 consensus_queue.cc:260] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.059348 12397 raft_consensus.cc:374] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:23.059311 12391 consensus_queue.cc:260] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.059409 12397 raft_consensus.cc:729] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27ca17f31e942acbabff2b9e6b96d63, State: Initialized, Role: FOLLOWER
I20260808 04:45:23.059849 12397 consensus_queue.cc:260] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.059927 12170 heartbeater.cc:503] Master 127.4.217.62:45045 was elected leader, sending a full tablet report...
I20260808 04:45:23.060321 12396 ts_tablet_manager.cc:1428] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:45:23.060576 12365 heartbeater.cc:503] Master 127.4.217.62:45045 was elected leader, sending a full tablet report...
I20260808 04:45:23.060798 12397 ts_tablet_manager.cc:1428] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260808 04:45:23.061590 12391 ts_tablet_manager.cc:1428] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 04:45:23.061937 11960 heartbeater.cc:503] Master 127.4.217.62:45045 was elected leader, sending a full tablet report...
W20260808 04:45:23.237747 12366 tablet.cc:2369] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:45:23.268337 12171 tablet.cc:2369] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:45:23.302531 11961 tablet.cc:2369] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:23.335306 12405 raft_consensus.cc:482] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:23.335562 12405 raft_consensus.cc:504] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.336762 12405 leader_election.cc:288] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a27ca17f31e942acbabff2b9e6b96d63 (127.4.217.3:46759), d48ed79f65664922933f0ff9a73eed2a (127.4.217.2:38955)
I20260808 04:45:23.341728 12402 raft_consensus.cc:482] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:23.341919 12402 raft_consensus.cc:504] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.342248 12409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.336976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:46759 (local address 127.4.217.1:41299)
0808 04:45:23.337337 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:23.337354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:23.337480 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:23.337826 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:23.337830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:23.337876 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:23.338190 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.338195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.339232 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.339236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.340852 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.340864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.341767 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.341772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.341874 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.341904 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:45:23.341989 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:45:23.342083 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":244,"spinlock_wait_cycles":11776,"thread_start_us":94,"threads_started":1}
I20260808 04:45:23.342249 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.337152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:41299 (local address 127.4.217.3:46759)
0808 04:45:23.337294 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.337299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.337456 (+ 157us) server_negotiation.cc:407] Connection header received
0808 04:45:23.337623 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.337627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.337690 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.337781 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:23.338322 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.339111 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.340987 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.341618 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.341745 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.341811 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.342019 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.342112 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:23.342377 12390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.337214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:49975 (local address 127.4.217.2:38955)
0808 04:45:23.337346 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.337353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.337583 (+ 230us) server_negotiation.cc:407] Connection header received
0808 04:45:23.337672 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.337674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.337735 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.337813 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:23.338322 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.339111 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.340987 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.341618 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.341745 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.341811 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.342186 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.342260 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:45:23.342782 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.337114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:38955 (local address 127.4.217.1:49975)
0808 04:45:23.337472 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:23.337485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:23.337587 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:23.337834 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:23.337837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:23.337873 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:23.338185 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.338194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.339232 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.339236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.340852 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.340864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.341892 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.341896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.342036 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.342064 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:45:23.342495 (+ 431us) client_negotiation.cc:240] Negotiation successful
0808 04:45:23.342594 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":243,"spinlock_wait_cycles":30848,"thread_start_us":105,"threads_started":1}
I20260808 04:45:23.342911 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d24f6cde0d41e4bc9b30e9aa59cad3" candidate_uuid: "2961fa5e16df418cbe8951556f5acee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27ca17f31e942acbabff2b9e6b96d63" is_pre_election: true
I20260808 04:45:23.343252 12402 leader_election.cc:288] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961fa5e16df418cbe8951556f5acee4 (127.4.217.1:44073), a27ca17f31e942acbabff2b9e6b96d63 (127.4.217.3:46759)
I20260808 04:45:23.343230 12302 raft_consensus.cc:2454] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961fa5e16df418cbe8951556f5acee4 in term 0.
I20260808 04:45:23.343564 12093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d24f6cde0d41e4bc9b30e9aa59cad3" candidate_uuid: "2961fa5e16df418cbe8951556f5acee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48ed79f65664922933f0ff9a73eed2a" is_pre_election: true
I20260808 04:45:23.343801 12093 raft_consensus.cc:2454] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961fa5e16df418cbe8951556f5acee4 in term 0.
I20260808 04:45:23.343775 11795 leader_election.cc:302] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961fa5e16df418cbe8951556f5acee4, a27ca17f31e942acbabff2b9e6b96d63; no voters:
I20260808 04:45:23.344117 12405 raft_consensus.cc:2781] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:23.344224 12405 raft_consensus.cc:482] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:23.344288 12405 raft_consensus.cc:3037] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:23.345737 12405 raft_consensus.cc:504] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.346131 12405 leader_election.cc:288] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [CANDIDATE]: Term 1 election: Requested vote from peers a27ca17f31e942acbabff2b9e6b96d63 (127.4.217.3:46759), d48ed79f65664922933f0ff9a73eed2a (127.4.217.2:38955)
I20260808 04:45:23.346489 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d24f6cde0d41e4bc9b30e9aa59cad3" candidate_uuid: "2961fa5e16df418cbe8951556f5acee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27ca17f31e942acbabff2b9e6b96d63"
I20260808 04:45:23.346522 12093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d24f6cde0d41e4bc9b30e9aa59cad3" candidate_uuid: "2961fa5e16df418cbe8951556f5acee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48ed79f65664922933f0ff9a73eed2a"
I20260808 04:45:23.346630 12093 raft_consensus.cc:3037] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:23.346619 12302 raft_consensus.cc:3037] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:23.347905 12093 raft_consensus.cc:2454] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961fa5e16df418cbe8951556f5acee4 in term 1.
I20260808 04:45:23.347906 12302 raft_consensus.cc:2454] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961fa5e16df418cbe8951556f5acee4 in term 1.
I20260808 04:45:23.348263 11797 leader_election.cc:302] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961fa5e16df418cbe8951556f5acee4, d48ed79f65664922933f0ff9a73eed2a; no voters:
I20260808 04:45:23.348423 12405 raft_consensus.cc:2781] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:23.348682 12405 raft_consensus.cc:686] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 1 LEADER]: Becoming Leader. State: Replica: 2961fa5e16df418cbe8951556f5acee4, State: Running, Role: LEADER
I20260808 04:45:23.349040 12405 consensus_queue.cc:237] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } }
I20260808 04:45:23.349277 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.344955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:39029 (local address 127.4.217.1:44073)
0808 04:45:23.345072 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.345077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.345103 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:45:23.345138 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.345140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.345176 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.345272 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:23.345531 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.346268 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.347887 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.348481 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.348588 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.348645 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.349102 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.349170 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 04:45:23.349277 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.345020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:37805 (local address 127.4.217.3:46759)
0808 04:45:23.345128 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.345133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.345148 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:23.345188 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.345191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.345238 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.345318 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:23.345708 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.346392 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.347901 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.348520 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.348601 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.348647 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.349102 (+ 455us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.349169 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 04:45:23.349645 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.343574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:44073 (local address 127.4.217.2:39029)
0808 04:45:23.343966 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:23.343982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:23.344179 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:23.345285 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:23.345289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:23.345306 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:23.345415 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.345420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.346398 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.346401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.347731 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.347744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.348750 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.348754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.348932 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.348972 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 04:45:23.349402 (+ 430us) client_negotiation.cc:240] Negotiation successful
0808 04:45:23.349486 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":41,"thread_start_us":110,"threads_started":1}
I20260808 04:45:23.349645 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.343963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:46759 (local address 127.4.217.2:37805)
0808 04:45:23.344241 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:23.344255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:23.344337 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:23.345423 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:23.345426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:23.345446 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:23.345577 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.345588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.346557 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.346560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.347770 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:23.347781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.348753 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.348755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:23.348928 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.348978 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 04:45:23.349448 (+ 470us) client_negotiation.cc:240] Negotiation successful
0808 04:45:23.349507 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":64,"thread_start_us":78,"threads_started":1}
I20260808 04:45:23.350119 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d24f6cde0d41e4bc9b30e9aa59cad3" candidate_uuid: "d48ed79f65664922933f0ff9a73eed2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27ca17f31e942acbabff2b9e6b96d63" is_pre_election: true
I20260808 04:45:23.350178 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d24f6cde0d41e4bc9b30e9aa59cad3" candidate_uuid: "d48ed79f65664922933f0ff9a73eed2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961fa5e16df418cbe8951556f5acee4" is_pre_election: true
I20260808 04:45:23.350322 12302 raft_consensus.cc:2379] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d48ed79f65664922933f0ff9a73eed2a in current term 1: Already voted for candidate 2961fa5e16df418cbe8951556f5acee4 in this term.
I20260808 04:45:23.350857 11998 leader_election.cc:302] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d48ed79f65664922933f0ff9a73eed2a; no voters: 2961fa5e16df418cbe8951556f5acee4, a27ca17f31e942acbabff2b9e6b96d63
I20260808 04:45:23.351107 12402 raft_consensus.cc:2726] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 04:45:23.352752 11662 catalog_manager.cc:5495] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2961fa5e16df418cbe8951556f5acee4 (127.4.217.1). New cstate: current_term: 1 leader_uuid: "2961fa5e16df418cbe8951556f5acee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961fa5e16df418cbe8951556f5acee4" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 44073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:23.396879 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.393207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.4.217.1:44073)
0808 04:45:23.393356 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.393360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.393373 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:23.393484 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.393486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.393549 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.393642 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:23.393875 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.394472 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.395372 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:23.395586 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:23.395668 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:23.395949 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:23.396387 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:23.396509 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.396630 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.396678 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:45:23.403016 12302 raft_consensus.cc:1261] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 1 FOLLOWER]: Refusing update from remote peer 2961fa5e16df418cbe8951556f5acee4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:23.403061 12093 raft_consensus.cc:1261] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 1 FOLLOWER]: Refusing update from remote peer 2961fa5e16df418cbe8951556f5acee4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:23.403723 12405 consensus_queue.cc:1035] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a27ca17f31e942acbabff2b9e6b96d63" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:23.403955 12414 consensus_queue.cc:1035] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d48ed79f65664922933f0ff9a73eed2a" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:23.421865 11662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:23.422094 11662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:23.424810 11662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ac37b18adeb4b3593200f2691e49f5f: Sending DeleteTablet for 3 replicas of tablet d4d24f6cde0d41e4bc9b30e9aa59cad3
I20260808 04:45:23.425731 12267 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:23 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:55142
I20260808 04:45:23.425732 11867 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:23 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:46856
I20260808 04:45:23.426292 12436 tablet_replica.cc:330] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4: stopping tablet replica
I20260808 04:45:23.426312 12437 tablet_replica.cc:330] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: stopping tablet replica
I20260808 04:45:23.426481 12057 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:23 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:38126
I20260808 04:45:23.426636 12437 raft_consensus.cc:2229] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:23.426811 12436 raft_consensus.cc:2229] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:23.426990 12437 raft_consensus.cc:2258] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:23.427320 12436 raft_consensus.cc:2258] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P 2961fa5e16df418cbe8951556f5acee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:23.427560 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 11765
I20260808 04:45:23.428745 12437 ts_tablet_manager.cc:1905] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:23.431559 12439 tablet_replica.cc:330] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: stopping tablet replica
I20260808 04:45:23.431895 12439 raft_consensus.cc:2229] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:23.431914 12437 ts_tablet_manager.cc:1918] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:23.432013 12437 log.cc:1196] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/d4d24f6cde0d41e4bc9b30e9aa59cad3
I20260808 04:45:23.432262 12439 raft_consensus.cc:2258] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:23.432350 12437 ts_tablet_manager.cc:1939] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P a27ca17f31e942acbabff2b9e6b96d63: Deleting consensus metadata
I20260808 04:45:23.433706 12439 ts_tablet_manager.cc:1905] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:23.436079 11643 catalog_manager.cc:4841] TS a27ca17f31e942acbabff2b9e6b96d63 (127.4.217.3:46759): tablet d4d24f6cde0d41e4bc9b30e9aa59cad3 (table test-table [id=3ce4c312e2a2479687a574d135dbfdf8]) successfully deleted
I20260808 04:45:23.436398 12439 ts_tablet_manager.cc:1918] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:23.436512 12439 log.cc:1196] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/d4d24f6cde0d41e4bc9b30e9aa59cad3
I20260808 04:45:23.437153 12439 ts_tablet_manager.cc:1939] T d4d24f6cde0d41e4bc9b30e9aa59cad3 P d48ed79f65664922933f0ff9a73eed2a: Deleting consensus metadata
I20260808 04:45:23.438248 11647 catalog_manager.cc:4841] TS d48ed79f65664922933f0ff9a73eed2a (127.4.217.2:38955): tablet d4d24f6cde0d41e4bc9b30e9aa59cad3 (table test-table [id=3ce4c312e2a2479687a574d135dbfdf8]) successfully deleted
I20260808 04:45:23.442247 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 11970
W20260808 04:45:23.442266 11647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:44073 (error 108)
I20260808 04:45:23.451305 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.450053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:44073 (local address 127.0.0.1:46874)
0808 04:45:23.451091 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:23.451186 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":899}
W20260808 04:45:23.451520 11647 catalog_manager.cc:4584] TS 2961fa5e16df418cbe8951556f5acee4 (127.4.217.1:44073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4d24f6cde0d41e4bc9b30e9aa59cad3: Network error: Client connection negotiation failed: client connection to 127.4.217.1:44073: connect: Connection refused (error 111)
I20260808 04:45:23.458952 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 12178
I20260808 04:45:23.475558 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 11614
2026-08-08T04:45:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2811 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T04:45:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:23Z Disabled control of system clock
I20260808 04:45:23.516533 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:45905
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:45905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 04:45:23.626796 12459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:23.627024 12459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:23.627059 12459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:23.630913 12459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:23.630959 12459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:23.630975 12459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:23.630990 12459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:23.631006 12459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:23.635509 12459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:45905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:45905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:23.636386 12459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:23.637769 12459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:23.646111 12481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:23.646124 12474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:23.646135 12459 server_base.cc:1034] running on GCE node
W20260808 04:45:23.646124 12475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:23.646915 12459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:23.647914 12459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:23.649080 12459 hybrid_clock.cc:648] HybridClock initialized: now 1786164323649066 us; error 29 us; skew 500 ppm
I20260808 04:45:23.650838 12459 webserver.cc:460] Webserver started at http://127.4.217.62:44119/ using document root <none> and password file <none>
I20260808 04:45:23.651401 12459 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:23.651470 12459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:23.651686 12459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:23.653704 12459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "a0e075afb4b5466c8e5b6800c3ede1c3"
format_stamp: "Formatted at 2026-08-08 04:45:23 on dist-test-slave-2kcd"
server_key: "725ca4216ca461002e3e07f2b9da92ab"
server_key_iv: "f8b7efcd31e5517b3ebbb6c5d79aff2d"
server_key_version: "encryptionkey@0"
I20260808 04:45:23.654263 12459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "a0e075afb4b5466c8e5b6800c3ede1c3"
format_stamp: "Formatted at 2026-08-08 04:45:23 on dist-test-slave-2kcd"
server_key: "725ca4216ca461002e3e07f2b9da92ab"
server_key_iv: "f8b7efcd31e5517b3ebbb6c5d79aff2d"
server_key_version: "encryptionkey@0"
I20260808 04:45:23.657814 12459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:23.660041 12488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:23.661201 12459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:23.661326 12459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "a0e075afb4b5466c8e5b6800c3ede1c3"
format_stamp: "Formatted at 2026-08-08 04:45:23 on dist-test-slave-2kcd"
server_key: "725ca4216ca461002e3e07f2b9da92ab"
server_key_iv: "f8b7efcd31e5517b3ebbb6c5d79aff2d"
server_key_version: "encryptionkey@0"
I20260808 04:45:23.661427 12459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:23.665836 12459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:23.666405 12459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:23.666599 12459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:23.673753 12459 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:45905
I20260808 04:45:23.673780 12565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:45905 every 8 connection(s)
I20260808 04:45:23.674628 12459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:23.677884 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:23.682017 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 12459
I20260808 04:45:23.682152 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:23.682420 4964 external_mini_cluster.cc:1428] Setting key 58768e0b468e4b2a04142dd893f0b881
I20260808 04:45:23.683825 12567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3: Bootstrap starting.
I20260808 04:45:23.686795 12567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:23.688047 12567 log.cc:824] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:23.688918 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.683823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.4.217.62:45905)
0808 04:45:23.684653 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.684675 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.684720 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:45:23.685676 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.685703 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.686197 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.686560 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:23.687396 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:23.687401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:23.687412 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:23.687422 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:23.687478 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:23.687494 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:23.687584 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.687732 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.688009 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":137,"threads_started":1}
I20260808 04:45:23.690124 12567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3: No bootstrap required, opened a new log
I20260808 04:45:23.693763 12567 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e075afb4b5466c8e5b6800c3ede1c3" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45905 } }
I20260808 04:45:23.693964 12567 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:23.693998 12567 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e075afb4b5466c8e5b6800c3ede1c3, State: Initialized, Role: FOLLOWER
I20260808 04:45:23.694368 12567 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e075afb4b5466c8e5b6800c3ede1c3" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45905 } }
I20260808 04:45:23.694527 12567 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:23.694620 12567 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:23.694726 12567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:23.695727 12567 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e075afb4b5466c8e5b6800c3ede1c3" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45905 } }
I20260808 04:45:23.696007 12567 leader_election.cc:302] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0e075afb4b5466c8e5b6800c3ede1c3; no voters:
I20260808 04:45:23.696244 12567 leader_election.cc:288] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:23.696352 12577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:23.696626 12577 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [term 1 LEADER]: Becoming Leader. State: Replica: a0e075afb4b5466c8e5b6800c3ede1c3, State: Running, Role: LEADER
I20260808 04:45:23.697008 12577 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e075afb4b5466c8e5b6800c3ede1c3" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45905 } }
I20260808 04:45:23.697544 12567 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:23.698721 12581 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0e075afb4b5466c8e5b6800c3ede1c3. Latest consensus state: current_term: 1 leader_uuid: "a0e075afb4b5466c8e5b6800c3ede1c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e075afb4b5466c8e5b6800c3ede1c3" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45905 } } }
I20260808 04:45:23.698841 12581 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:23.699153 12579 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0e075afb4b5466c8e5b6800c3ede1c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e075afb4b5466c8e5b6800c3ede1c3" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 45905 } } }
I20260808 04:45:23.699334 12579 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:23.699246 12590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:23.702459 12590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:23.707669 12590 catalog_manager.cc:1347] Generated new cluster ID: dcc250f5bb104198b998898b23600540
I20260808 04:45:23.707741 12590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:23.748848 12590 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:23.749226 12590 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:23.762472 12590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3: Generated new TSK 0
I20260808 04:45:23.763177 12590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:45:23.777204 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45905
--builtin_ntp_servers=127.4.217.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 04:45:23.904367 12610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:23.904629 12610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:23.904670 12610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:23.908674 12610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:23.908733 12610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:23.908851 12610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:23.913555 12610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:23.914568 12610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:23.915885 12610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:23.925982 12622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:23.926003 12620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:23.926074 12610 server_base.cc:1034] running on GCE node
W20260808 04:45:23.926054 12618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:23.927572 12610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:23.928133 12610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:23.929301 12610 hybrid_clock.cc:648] HybridClock initialized: now 1786164323929265 us; error 50 us; skew 500 ppm
I20260808 04:45:23.931221 12610 webserver.cc:460] Webserver started at http://127.4.217.1:46715/ using document root <none> and password file <none>
I20260808 04:45:23.931804 12610 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:23.931867 12610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:23.932107 12610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:23.934224 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "4e7b4fa5207a4bb0af7c48fec5034c27"
format_stamp: "Formatted at 2026-08-08 04:45:23 on dist-test-slave-2kcd"
server_key: "dfe5977fe89b963b407141d4e376ff21"
server_key_iv: "97a0a03ae7e3eeefb3fcdc2d5827234c"
server_key_version: "encryptionkey@0"
I20260808 04:45:23.934787 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "4e7b4fa5207a4bb0af7c48fec5034c27"
format_stamp: "Formatted at 2026-08-08 04:45:23 on dist-test-slave-2kcd"
server_key: "dfe5977fe89b963b407141d4e376ff21"
server_key_iv: "97a0a03ae7e3eeefb3fcdc2d5827234c"
server_key_version: "encryptionkey@0"
I20260808 04:45:23.938458 12610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 04:45:23.940872 12632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:23.942085 12610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:23.942235 12610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "4e7b4fa5207a4bb0af7c48fec5034c27"
format_stamp: "Formatted at 2026-08-08 04:45:23 on dist-test-slave-2kcd"
server_key: "dfe5977fe89b963b407141d4e376ff21"
server_key_iv: "97a0a03ae7e3eeefb3fcdc2d5827234c"
server_key_version: "encryptionkey@0"
I20260808 04:45:23.942351 12610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:23.946967 12610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:23.947566 12610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:23.947748 12610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:23.948283 12610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:23.949285 12610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:23.949359 12610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:23.949424 12610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:23.949468 12610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:23.958714 12610 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:34995
I20260808 04:45:23.958761 12803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:34995 every 8 connection(s)
I20260808 04:45:23.959631 12610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:23.965440 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 12610
I20260808 04:45:23.965579 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:23.965896 4964 external_mini_cluster.cc:1428] Setting key f5cfbd55c2b1bc116a5b6bfec95cd50b
I20260808 04:45:23.965935 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.961482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:33935 (local address 127.4.217.62:45905)
0808 04:45:23.961637 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:23.961643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:23.962735 (+ 1092us) server_negotiation.cc:407] Connection header received
0808 04:45:23.963982 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:23.963985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:23.964029 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:23.964105 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:23.965320 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:23.965325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:23.965328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:23.965342 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:23.965369 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:23.965383 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:23.965520 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:23.965721 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 04:45:23.965774 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:23.966830 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:23.961933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:45905 (local address 127.4.217.1:33935)
0808 04:45:23.962587 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:23.962627 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:23.963798 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:23.964267 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:23.964273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:23.964397 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:23.965030 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:23.965032 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:23.965073 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:23.965143 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:23.965516 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:23.965527 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:23.965537 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 04:45:23.965731 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 04:45:23.965965 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260808 04:45:23.968298 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45905
--builtin_ntp_servers=127.4.217.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 04:45:23.968340 12804 heartbeater.cc:347] Connected to a master server at 127.4.217.62:45905
I20260808 04:45:23.968590 12804 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:23.969107 12804 heartbeater.cc:511] Master 127.4.217.62:45905 requested a full tablet report, sending...
I20260808 04:45:23.970849 12516 ts_manager.cc:194] Registered new tserver with Master: 4e7b4fa5207a4bb0af7c48fec5034c27 (127.4.217.1:34995)
W20260808 04:45:24.085621 12807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:24.085902 12807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:24.085983 12807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:24.089970 12807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:24.090044 12807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:24.090134 12807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:24.094812 12807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:24.095880 12807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:24.097280 12807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:24.107124 12814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:24.107134 12813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:24.107189 12807 server_base.cc:1034] running on GCE node
W20260808 04:45:24.107103 12820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:24.107826 12807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:24.108436 12807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:24.109620 12807 hybrid_clock.cc:648] HybridClock initialized: now 1786164324109583 us; error 50 us; skew 500 ppm
I20260808 04:45:24.111346 12807 webserver.cc:460] Webserver started at http://127.4.217.2:43697/ using document root <none> and password file <none>
I20260808 04:45:24.111917 12807 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:24.112012 12807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:24.112239 12807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:24.114264 12807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "526b3096f50c447a961bd3f0cf4a1695"
format_stamp: "Formatted at 2026-08-08 04:45:24 on dist-test-slave-2kcd"
server_key: "929a7e5e845a0d029652ef517839eb6a"
server_key_iv: "59fc3bfafd11b62f69e9ac088f87c78b"
server_key_version: "encryptionkey@0"
I20260808 04:45:24.114825 12807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "526b3096f50c447a961bd3f0cf4a1695"
format_stamp: "Formatted at 2026-08-08 04:45:24 on dist-test-slave-2kcd"
server_key: "929a7e5e845a0d029652ef517839eb6a"
server_key_iv: "59fc3bfafd11b62f69e9ac088f87c78b"
server_key_version: "encryptionkey@0"
I20260808 04:45:24.118438 12807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 04:45:24.120775 12826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:24.121971 12807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:24.122116 12807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "526b3096f50c447a961bd3f0cf4a1695"
format_stamp: "Formatted at 2026-08-08 04:45:24 on dist-test-slave-2kcd"
server_key: "929a7e5e845a0d029652ef517839eb6a"
server_key_iv: "59fc3bfafd11b62f69e9ac088f87c78b"
server_key_version: "encryptionkey@0"
I20260808 04:45:24.122227 12807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:24.126585 12807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:24.127157 12807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:24.127341 12807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:24.127869 12807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:24.128753 12807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:24.128880 12807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:24.128952 12807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:24.129005 12807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:24.137813 12807 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:45021
I20260808 04:45:24.137881 12993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:45021 every 8 connection(s)
I20260808 04:45:24.138724 12807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:24.144775 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.140582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:52153 (local address 127.4.217.62:45905)
0808 04:45:24.140724 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.140728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.141731 (+ 1003us) server_negotiation.cc:407] Connection header received
0808 04:45:24.143031 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.143043 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.143092 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.143203 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.144261 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.144264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.144267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.144280 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.144309 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.144318 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.144416 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.144589 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.144637 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:45:24.145587 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.140943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:45905 (local address 127.4.217.2:52153)
0808 04:45:24.141590 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:24.141628 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:24.142822 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:24.143370 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:24.143376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:24.143458 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:24.144007 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:24.144008 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:24.144048 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:24.144113 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:24.144410 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:24.144420 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:24.144429 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:45:24.144626 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 04:45:24.144883 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260808 04:45:24.145961 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 12807
I20260808 04:45:24.146142 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:24.146507 4964 external_mini_cluster.cc:1428] Setting key b8b05474ae702728bc78c57b5213c140
I20260808 04:45:24.146903 12995 heartbeater.cc:347] Connected to a master server at 127.4.217.62:45905
I20260808 04:45:24.147150 12995 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:24.147552 12995 heartbeater.cc:511] Master 127.4.217.62:45905 requested a full tablet report, sending...
I20260808 04:45:24.148578 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45905
--builtin_ntp_servers=127.4.217.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 04:45:24.148658 12516 ts_manager.cc:194] Registered new tserver with Master: 526b3096f50c447a961bd3f0cf4a1695 (127.4.217.2:45021)
W20260808 04:45:24.263238 13001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:24.263477 13001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:24.263516 13001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:24.267755 13001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:24.267812 13001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:24.267871 13001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:24.272490 13001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:24.273531 13001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:24.274909 13001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:24.284477 13014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:24.284559 13018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:24.284709 13001 server_base.cc:1034] running on GCE node
W20260808 04:45:24.284498 13015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:24.285223 13001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:24.285840 13001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:24.287043 13001 hybrid_clock.cc:648] HybridClock initialized: now 1786164324287020 us; error 25 us; skew 500 ppm
I20260808 04:45:24.288857 13001 webserver.cc:460] Webserver started at http://127.4.217.3:44499/ using document root <none> and password file <none>
I20260808 04:45:24.289431 13001 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:24.289526 13001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:24.289762 13001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:24.291776 13001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "dc613925169b4d46a44071330192328d"
format_stamp: "Formatted at 2026-08-08 04:45:24 on dist-test-slave-2kcd"
server_key: "269e9b1a13cec80a832fcf92baad6a3b"
server_key_iv: "2aac6e908e03629f1583e5a99b3a6a01"
server_key_version: "encryptionkey@0"
I20260808 04:45:24.292368 13001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "dc613925169b4d46a44071330192328d"
format_stamp: "Formatted at 2026-08-08 04:45:24 on dist-test-slave-2kcd"
server_key: "269e9b1a13cec80a832fcf92baad6a3b"
server_key_iv: "2aac6e908e03629f1583e5a99b3a6a01"
server_key_version: "encryptionkey@0"
I20260808 04:45:24.296171 13001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:45:24.298732 13024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:24.300050 13001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:45:24.300189 13001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "dc613925169b4d46a44071330192328d"
format_stamp: "Formatted at 2026-08-08 04:45:24 on dist-test-slave-2kcd"
server_key: "269e9b1a13cec80a832fcf92baad6a3b"
server_key_iv: "2aac6e908e03629f1583e5a99b3a6a01"
server_key_version: "encryptionkey@0"
I20260808 04:45:24.300300 13001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:24.304883 13001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:24.305472 13001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:24.305665 13001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:24.306216 13001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:24.307121 13001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:24.307169 13001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:24.307206 13001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:24.307236 13001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:24.316368 13001 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:35315
I20260808 04:45:24.316408 13205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:35315 every 8 connection(s)
I20260808 04:45:24.317307 13001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:24.323458 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.319123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:54935 (local address 127.4.217.62:45905)
0808 04:45:24.319277 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.319281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.320264 (+ 983us) server_negotiation.cc:407] Connection header received
0808 04:45:24.321713 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.321717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.321764 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.321876 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.322958 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.322962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.322965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.322984 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.323003 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.323011 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.323117 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.323269 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.323318 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:45:24.324239 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.319467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:45905 (local address 127.4.217.3:54935)
0808 04:45:24.320117 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:24.320157 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:24.321487 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:24.322044 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:24.322050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:24.322134 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:24.322706 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:24.322707 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:24.322746 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:24.322809 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:24.323102 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:24.323111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:24.323117 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:45:24.323299 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 04:45:24.323524 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260808 04:45:24.325361 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 13001
I20260808 04:45:24.325484 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:24.325567 13206 heartbeater.cc:347] Connected to a master server at 127.4.217.62:45905
I20260808 04:45:24.325830 4964 external_mini_cluster.cc:1428] Setting key 0cb4b13039e4e220a905e5b890874011
I20260808 04:45:24.325845 13206 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:24.326326 13206 heartbeater.cc:511] Master 127.4.217.62:45905 requested a full tablet report, sending...
I20260808 04:45:24.327486 12516 ts_manager.cc:194] Registered new tserver with Master: dc613925169b4d46a44071330192328d (127.4.217.3:35315)
I20260808 04:45:24.328366 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:45:24.331810 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.330338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.4.217.62:45905)
0808 04:45:24.330479 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.330484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.330624 (+ 140us) server_negotiation.cc:407] Connection header received
0808 04:45:24.330715 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.330717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.330747 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.330809 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.331467 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.331469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.331471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.331479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.331491 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.331498 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.331575 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.331617 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.331645 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:45:24.334719 12515 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:24.337667 12515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:24.349835 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.346645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.4.217.3:35315)
0808 04:45:24.347104 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.347117 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.347534 (+ 417us) server_negotiation.cc:407] Connection header received
0808 04:45:24.347596 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.347601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.347769 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.347918 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.349148 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.349156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.349168 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.349181 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.349225 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.349255 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.349357 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.349554 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.349633 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":142,"threads_started":1}
I20260808 04:45:24.350121 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.346198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.4.217.2:45021)
0808 04:45:24.346876 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.346891 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.347328 (+ 437us) server_negotiation.cc:407] Connection header received
0808 04:45:24.347559 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.347568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.347768 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.347879 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.349472 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.349478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.349489 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.349502 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.349554 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.349592 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.349692 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.349879 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.349955 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":120,"threads_started":1}
I20260808 04:45:24.350142 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.346863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:34995 (local address 127.0.0.1:47788)
0808 04:45:24.347552 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:24.347566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:24.347629 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:24.348597 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:24.348603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:24.348619 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:24.349460 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:24.349462 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:24.349472 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:24.349498 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:24.349892 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:24.349894 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:24.349897 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 04:45:24.349968 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:45:24.350015 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":93,"threads_started":1}
I20260808 04:45:24.350524 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.346309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:35315 (local address 127.0.0.1:57430)
0808 04:45:24.347313 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:24.347327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:24.347421 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:24.347920 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:24.347927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:24.347998 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:24.348582 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:24.348585 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:24.348606 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:24.348675 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:24.349360 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:24.349373 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:24.349382 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:45:24.349536 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 04:45:24.350382 (+ 846us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"spinlock_wait_cycles":8576,"thread_start_us":106,"threads_started":1}
I20260808 04:45:24.350560 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.347148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.4.217.1:34995)
0808 04:45:24.347868 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.347880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.347902 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:45:24.347952 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.347956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.348107 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.348236 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.349645 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.349653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.349663 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.349676 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.349715 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.349748 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.349850 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.350304 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.350372 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":105,"threads_started":1}
I20260808 04:45:24.350804 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.346002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:45021 (local address 127.0.0.1:55820)
0808 04:45:24.347183 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:24.347234 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:24.347417 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:24.348608 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:24.348611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:24.348626 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:24.349270 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:24.349272 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:24.349281 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:24.349326 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:24.349737 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:24.349740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:24.349743 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 04:45:24.350496 (+ 753us) client_negotiation.cc:240] Negotiation successful
0808 04:45:24.350534 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":104,"threads_started":1}
I20260808 04:45:24.353291 12703 tablet_service.cc:1467] Processing CreateTablet for tablet ea2862546d1a4b87877eb8e39460bbda (DEFAULT_TABLE table=test-table [id=dae4c114fc95469ea796b9211d7c57ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:24.353406 13101 tablet_service.cc:1467] Processing CreateTablet for tablet ea2862546d1a4b87877eb8e39460bbda (DEFAULT_TABLE table=test-table [id=dae4c114fc95469ea796b9211d7c57ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:24.353405 12897 tablet_service.cc:1467] Processing CreateTablet for tablet ea2862546d1a4b87877eb8e39460bbda (DEFAULT_TABLE table=test-table [id=dae4c114fc95469ea796b9211d7c57ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:24.354369 12703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2862546d1a4b87877eb8e39460bbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:24.354422 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2862546d1a4b87877eb8e39460bbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:24.354423 12897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2862546d1a4b87877eb8e39460bbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:24.360467 13232 tablet_bootstrap.cc:492] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27: Bootstrap starting.
I20260808 04:45:24.361903 13234 tablet_bootstrap.cc:492] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: Bootstrap starting.
I20260808 04:45:24.363759 13232 tablet_bootstrap.cc:654] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:24.364089 13234 tablet_bootstrap.cc:654] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:24.364281 13235 tablet_bootstrap.cc:492] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: Bootstrap starting.
I20260808 04:45:24.364763 13232 log.cc:824] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:24.364862 13234 log.cc:824] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:24.366477 13235 tablet_bootstrap.cc:654] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:24.366917 13234 tablet_bootstrap.cc:492] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: No bootstrap required, opened a new log
I20260808 04:45:24.367031 13232 tablet_bootstrap.cc:492] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27: No bootstrap required, opened a new log
I20260808 04:45:24.367121 13234 ts_tablet_manager.cc:1397] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:45:24.367228 13232 ts_tablet_manager.cc:1397] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260808 04:45:24.367321 13235 log.cc:824] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:24.369352 13235 tablet_bootstrap.cc:492] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: No bootstrap required, opened a new log
I20260808 04:45:24.369535 13235 ts_tablet_manager.cc:1397] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:45:24.370929 13232 raft_consensus.cc:348] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } }
I20260808 04:45:24.371167 13232 raft_consensus.cc:374] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:24.371223 13232 raft_consensus.cc:729] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7b4fa5207a4bb0af7c48fec5034c27, State: Initialized, Role: FOLLOWER
I20260808 04:45:24.371697 13232 consensus_queue.cc:260] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } }
I20260808 04:45:24.371717 13234 raft_consensus.cc:348] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } }
I20260808 04:45:24.371960 13234 raft_consensus.cc:374] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:24.372025 13234 raft_consensus.cc:729] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc613925169b4d46a44071330192328d, State: Initialized, Role: FOLLOWER
I20260808 04:45:24.372437 12804 heartbeater.cc:503] Master 127.4.217.62:45905 was elected leader, sending a full tablet report...
I20260808 04:45:24.372571 13234 consensus_queue.cc:260] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } }
I20260808 04:45:24.372732 13232 ts_tablet_manager.cc:1428] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260808 04:45:24.373297 13206 heartbeater.cc:503] Master 127.4.217.62:45905 was elected leader, sending a full tablet report...
I20260808 04:45:24.373555 13234 ts_tablet_manager.cc:1428] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260808 04:45:24.374056 13235 raft_consensus.cc:348] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } }
I20260808 04:45:24.374298 13235 raft_consensus.cc:374] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:24.374367 13235 raft_consensus.cc:729] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526b3096f50c447a961bd3f0cf4a1695, State: Initialized, Role: FOLLOWER
I20260808 04:45:24.374761 13235 consensus_queue.cc:260] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } }
I20260808 04:45:24.375468 12995 heartbeater.cc:503] Master 127.4.217.62:45905 was elected leader, sending a full tablet report...
I20260808 04:45:24.375762 13235 ts_tablet_manager.cc:1428] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260808 04:45:24.389667 12996 tablet.cc:2369] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:24.431205 13240 raft_consensus.cc:482] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:24.431447 13240 raft_consensus.cc:504] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } }
I20260808 04:45:24.432678 13240 leader_election.cc:288] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 526b3096f50c447a961bd3f0cf4a1695 (127.4.217.2:45021), dc613925169b4d46a44071330192328d (127.4.217.3:35315)
I20260808 04:45:24.434775 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.432951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:45021 (local address 127.4.217.1:41589)
0808 04:45:24.433130 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:24.433147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:24.433236 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:24.433547 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:24.433549 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:24.433563 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:24.434109 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:24.434112 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:24.434129 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:24.434162 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:24.434482 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:24.434488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:24.434492 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:45:24.434584 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:45:24.434624 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 04:45:24.434827 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.433026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:41589 (local address 127.4.217.2:45021)
0808 04:45:24.433171 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.433176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.433193 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:45:24.433367 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.433370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.433418 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.433515 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.434307 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.434313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.434317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.434334 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.434355 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.434365 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.434467 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.434607 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.434667 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:24.434985 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.432952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:35315 (local address 127.4.217.1:46535)
0808 04:45:24.433521 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:24.433546 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:24.433634 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:24.433906 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:24.433909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:24.433925 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:24.434347 (+ 422us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:24.434349 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:24.434359 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:24.434386 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:24.434692 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:24.434695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:24.434699 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:45:24.434784 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:45:24.434817 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":128,"thread_start_us":151,"threads_started":1}
I20260808 04:45:24.434993 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.433085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:46535 (local address 127.4.217.3:35315)
0808 04:45:24.433223 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.433228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.433612 (+ 384us) server_negotiation.cc:407] Connection header received
0808 04:45:24.433750 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.433753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.433797 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.433885 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.434537 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.434542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.434548 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.434566 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.434591 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.434602 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.434689 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.434801 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.434849 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:45:24.435659 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2862546d1a4b87877eb8e39460bbda" candidate_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526b3096f50c447a961bd3f0cf4a1695" is_pre_election: true
I20260808 04:45:24.435660 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2862546d1a4b87877eb8e39460bbda" candidate_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc613925169b4d46a44071330192328d" is_pre_election: true
I20260808 04:45:24.435961 13130 raft_consensus.cc:2454] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7b4fa5207a4bb0af7c48fec5034c27 in term 0.
I20260808 04:45:24.435967 12920 raft_consensus.cc:2454] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7b4fa5207a4bb0af7c48fec5034c27 in term 0.
I20260808 04:45:24.436462 12633 leader_election.cc:302] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7b4fa5207a4bb0af7c48fec5034c27, 526b3096f50c447a961bd3f0cf4a1695; no voters:
I20260808 04:45:24.436775 13240 raft_consensus.cc:2781] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:24.436923 13240 raft_consensus.cc:482] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:24.436980 13240 raft_consensus.cc:3037] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:24.438059 13240 raft_consensus.cc:504] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } }
I20260808 04:45:24.438455 13240 leader_election.cc:288] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [CANDIDATE]: Term 1 election: Requested vote from peers 526b3096f50c447a961bd3f0cf4a1695 (127.4.217.2:45021), dc613925169b4d46a44071330192328d (127.4.217.3:35315)
I20260808 04:45:24.438822 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2862546d1a4b87877eb8e39460bbda" candidate_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc613925169b4d46a44071330192328d"
I20260808 04:45:24.438850 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2862546d1a4b87877eb8e39460bbda" candidate_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526b3096f50c447a961bd3f0cf4a1695"
I20260808 04:45:24.438975 13130 raft_consensus.cc:3037] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:24.439011 12920 raft_consensus.cc:3037] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:24.440506 13130 raft_consensus.cc:2454] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7b4fa5207a4bb0af7c48fec5034c27 in term 1.
I20260808 04:45:24.440555 12920 raft_consensus.cc:2454] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7b4fa5207a4bb0af7c48fec5034c27 in term 1.
I20260808 04:45:24.440937 12636 leader_election.cc:302] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7b4fa5207a4bb0af7c48fec5034c27, dc613925169b4d46a44071330192328d; no voters:
I20260808 04:45:24.441118 13240 raft_consensus.cc:2781] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:24.441336 13240 raft_consensus.cc:686] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [term 1 LEADER]: Becoming Leader. State: Replica: 4e7b4fa5207a4bb0af7c48fec5034c27, State: Running, Role: LEADER
I20260808 04:45:24.441684 13240 consensus_queue.cc:237] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } }
I20260808 04:45:24.445135 12513 catalog_manager.cc:5495] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e7b4fa5207a4bb0af7c48fec5034c27 (127.4.217.1). New cstate: current_term: 1 leader_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e7b4fa5207a4bb0af7c48fec5034c27" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34995 } health_report { overall_health: HEALTHY } } }
W20260808 04:45:24.460932 12805 tablet.cc:2369] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:24.471204 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.469694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.4.217.1:34995)
0808 04:45:24.469834 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.469838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.469853 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:24.469928 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.469930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.469987 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.470069 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.470814 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.470817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.470818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.470827 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.470840 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.470847 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.470915 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.470982 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.471015 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:24.477113 12920 raft_consensus.cc:1261] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [term 1 FOLLOWER]: Refusing update from remote peer 4e7b4fa5207a4bb0af7c48fec5034c27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:24.477113 13130 raft_consensus.cc:1261] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [term 1 FOLLOWER]: Refusing update from remote peer 4e7b4fa5207a4bb0af7c48fec5034c27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:24.477869 13252 consensus_queue.cc:1035] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526b3096f50c447a961bd3f0cf4a1695" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:24.478085 13240 consensus_queue.cc:1035] T ea2862546d1a4b87877eb8e39460bbda P 4e7b4fa5207a4bb0af7c48fec5034c27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc613925169b4d46a44071330192328d" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:24.486717 13256 mvcc.cc:205] Tried to move back new op lower bound from 7316129073050501120 to 7316129072915795968. Current Snapshot: MvccSnapshot[applied={T|T < 7316129073050501120}]
I20260808 04:45:24.496614 12515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:24.496789 12515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:24.499533 12515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0e075afb4b5466c8e5b6800c3ede1c3: Sending DeleteTablet for 3 replicas of tablet ea2862546d1a4b87877eb8e39460bbda
I20260808 04:45:24.500412 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2862546d1a4b87877eb8e39460bbda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:24 UTC) from {username='slave'} at 127.0.0.1:57430
I20260808 04:45:24.500458 12897 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2862546d1a4b87877eb8e39460bbda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:24 UTC) from {username='slave'} at 127.0.0.1:55820
I20260808 04:45:24.500941 13275 tablet_replica.cc:330] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: stopping tablet replica
I20260808 04:45:24.501183 12703 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2862546d1a4b87877eb8e39460bbda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:24 UTC) from {username='slave'} at 127.0.0.1:47788
I20260808 04:45:24.501307 13275 raft_consensus.cc:2229] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:24.501358 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 12610
I20260808 04:45:24.501660 13275 raft_consensus.cc:2258] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:24.504478 13276 tablet_replica.cc:330] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: stopping tablet replica
I20260808 04:45:24.504858 13276 raft_consensus.cc:2229] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:24.505159 13276 raft_consensus.cc:2258] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:24.507678 13276 ts_tablet_manager.cc:1905] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:24.510999 13276 ts_tablet_manager.cc:1918] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:24.511130 13276 log.cc:1196] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/ea2862546d1a4b87877eb8e39460bbda
I20260808 04:45:24.511437 13276 ts_tablet_manager.cc:1939] T ea2862546d1a4b87877eb8e39460bbda P 526b3096f50c447a961bd3f0cf4a1695: Deleting consensus metadata
I20260808 04:45:24.512997 12489 catalog_manager.cc:4841] TS 526b3096f50c447a961bd3f0cf4a1695 (127.4.217.2:45021): tablet ea2862546d1a4b87877eb8e39460bbda (table test-table [id=dae4c114fc95469ea796b9211d7c57ab]) successfully deleted
I20260808 04:45:24.513343 13275 ts_tablet_manager.cc:1905] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:24.515561 13275 ts_tablet_manager.cc:1918] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:24.515682 13275 log.cc:1196] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/ea2862546d1a4b87877eb8e39460bbda
I20260808 04:45:24.516031 13275 ts_tablet_manager.cc:1939] T ea2862546d1a4b87877eb8e39460bbda P dc613925169b4d46a44071330192328d: Deleting consensus metadata
I20260808 04:45:24.517160 12492 catalog_manager.cc:4841] TS dc613925169b4d46a44071330192328d (127.4.217.3:35315): tablet ea2862546d1a4b87877eb8e39460bbda (table test-table [id=dae4c114fc95469ea796b9211d7c57ab]) successfully deleted
W20260808 04:45:24.518208 12489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:34995 (error 108)
I20260808 04:45:24.519070 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 12807
I20260808 04:45:24.519340 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.518983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:34995 (local address 127.0.0.1:47818)
0808 04:45:24.519139 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:24.519240 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 04:45:24.521082 12489 catalog_manager.cc:4584] TS 4e7b4fa5207a4bb0af7c48fec5034c27 (127.4.217.1:34995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea2862546d1a4b87877eb8e39460bbda: Network error: Client connection negotiation failed: client connection to 127.4.217.1:34995: connect: Connection refused (error 111)
I20260808 04:45:24.531903 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 13001
I20260808 04:45:24.546193 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 12459
2026-08-08T04:45:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1069 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 04:45:24.576680 4964 mini_oidc.cc:152] Starting JWKS server
I20260808 04:45:24.578361 4964 webserver.cc:460] Webserver started at https://127.0.0.1:36883/ using document root <none> and password file <none>
I20260808 04:45:24.578497 4964 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:45:24.578734 4964 webserver.cc:460] Webserver started at http://127.0.0.1:41077/ using document root <none> and password file <none>
2026-08-08T04:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:24Z Disabled control of system clock
I20260808 04:45:24.589048 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35693
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35693
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36883/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/testchainca.cert with env {}
W20260808 04:45:24.702147 13286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:24.702378 13286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:24.702415 13286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:24.706344 13286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:24.706398 13286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:24.706413 13286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:24.706429 13286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:24.706444 13286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:45:24.706459 13286 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:45:24.706475 13286 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36883/jwks.json
I20260808 04:45:24.711678 13286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36883/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:24.712589 13286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:24.713523 13286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:24.722229 13294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:24.722204 13296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:24.722307 13293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:24.722368 13286 server_base.cc:1034] running on GCE node
I20260808 04:45:24.723069 13286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:24.724140 13286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:24.725302 13286 hybrid_clock.cc:648] HybridClock initialized: now 1786164324725283 us; error 30 us; skew 500 ppm
I20260808 04:45:24.727098 13286 webserver.cc:460] Webserver started at http://127.4.217.62:39003/ using document root <none> and password file <none>
I20260808 04:45:24.727615 13286 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:24.727676 13286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:24.727841 13286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:24.729935 13286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "88470d00d45e4a59b1bcb781de64efb5"
format_stamp: "Formatted at 2026-08-08 04:45:24 on dist-test-slave-2kcd"
server_key: "7f1ff5a81d533e8b1281f687f7b51ce7"
server_key_iv: "b08dfc2ccb357b23cfc4c70faeb00a43"
server_key_version: "encryptionkey@0"
I20260808 04:45:24.730469 13286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "88470d00d45e4a59b1bcb781de64efb5"
format_stamp: "Formatted at 2026-08-08 04:45:24 on dist-test-slave-2kcd"
server_key: "7f1ff5a81d533e8b1281f687f7b51ce7"
server_key_iv: "b08dfc2ccb357b23cfc4c70faeb00a43"
server_key_version: "encryptionkey@0"
I20260808 04:45:24.734287 13286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:45:24.736636 13306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:24.737888 13286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:24.738035 13286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "88470d00d45e4a59b1bcb781de64efb5"
format_stamp: "Formatted at 2026-08-08 04:45:24 on dist-test-slave-2kcd"
server_key: "7f1ff5a81d533e8b1281f687f7b51ce7"
server_key_iv: "b08dfc2ccb357b23cfc4c70faeb00a43"
server_key_version: "encryptionkey@0"
I20260808 04:45:24.738145 13286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:24.781507 13286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:24.782289 13286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:24.782500 13286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:24.789605 13286 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:35693
I20260808 04:45:24.789623 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:35693 every 8 connection(s)
I20260808 04:45:24.790558 13286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:24.793503 13383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:24.795001 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 13286
I20260808 04:45:24.795149 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:24.795413 4964 external_mini_cluster.cc:1428] Setting key 5535df82377914a138abdcaddd9f36cd
I20260808 04:45:24.799782 13383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5: Bootstrap starting.
I20260808 04:45:24.802475 13383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:24.803241 13383 log.cc:824] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:24.804487 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.796702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.4.217.62:35693)
0808 04:45:24.797377 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.797393 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.797428 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:45:24.798059 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.798074 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.798449 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.798744 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:24.799103 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:24.800122 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:24.801334 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:24.801782 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:24.801900 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:24.802757 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:24.802763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:24.802776 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:24.803161 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:24.803217 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:24.803237 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:24.803324 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:24.803523 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 04:45:24.803809 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":134,"threads_started":1}
I20260808 04:45:24.805344 13383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5: No bootstrap required, opened a new log
I20260808 04:45:24.807214 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:45:24.809285 13383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88470d00d45e4a59b1bcb781de64efb5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35693 } }
I20260808 04:45:24.809485 13383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:24.809546 13383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88470d00d45e4a59b1bcb781de64efb5, State: Initialized, Role: FOLLOWER
I20260808 04:45:24.809978 13383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88470d00d45e4a59b1bcb781de64efb5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35693 } }
I20260808 04:45:24.810108 13383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:24.810195 13383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:24.810334 13383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:24.811187 13383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88470d00d45e4a59b1bcb781de64efb5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35693 } }
I20260808 04:45:24.811512 13383 leader_election.cc:302] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88470d00d45e4a59b1bcb781de64efb5; no voters:
I20260808 04:45:24.811803 13383 leader_election.cc:288] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:24.811956 13391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:24.812211 13391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [term 1 LEADER]: Becoming Leader. State: Replica: 88470d00d45e4a59b1bcb781de64efb5, State: Running, Role: LEADER
I20260808 04:45:24.812515 13391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88470d00d45e4a59b1bcb781de64efb5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35693 } }
I20260808 04:45:24.813024 13383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:24.814312 13393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88470d00d45e4a59b1bcb781de64efb5. Latest consensus state: current_term: 1 leader_uuid: "88470d00d45e4a59b1bcb781de64efb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88470d00d45e4a59b1bcb781de64efb5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35693 } } }
I20260808 04:45:24.814445 13393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:24.814303 13392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88470d00d45e4a59b1bcb781de64efb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88470d00d45e4a59b1bcb781de64efb5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35693 } } }
I20260808 04:45:24.814587 13392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:24.814910 13403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:24.818001 13403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:24.823249 13403 catalog_manager.cc:1347] Generated new cluster ID: 1834995a73de4e959600e2b0532ec4b1
I20260808 04:45:24.823313 13403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:24.862036 13403 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:24.863095 13403 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:24.875221 13403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88470d00d45e4a59b1bcb781de64efb5: Generated new TSK 0
I20260808 04:45:24.875959 13403 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 04:45:24.917188 13425 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 04:45:24.917333 13425 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 04:45:24.917832 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:24.894172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.4.217.62:35693)
0808 04:45:24.894328 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:24.894336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:24.894539 (+ 203us) server_negotiation.cc:407] Connection header received
0808 04:45:24.894760 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:24.894762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:24.894812 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:24.894899 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:45:24.895211 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:24.895713 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:24.896988 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:24.897204 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:24.897288 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:24.897664 (+ 376us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:45:24.917282 (+ 19618us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 04:45:24.917695 (+ 413us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55522: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 04:45:24.918478 13387 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55522: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 04:45:24.919354 13424 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:24.894086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35693 (local address 127.0.0.1:55522)
0808 04:45:24.894447 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:24.894462 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:24.894600 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:24.894952 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:24.894956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:24.894965 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:45:24.895102 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:24.895109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:24.895841 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:24.895844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:24.896861 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:24.896871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:24.897318 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:24.897322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:24.897419 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:24.897471 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:45:24.917669 (+ 20198us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 04:45:24.917937 (+ 268us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.217.62:35693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.217.62:35693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 04:45:24.920295 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 13286
2026-08-08T04:45:24Z chronyd exiting
I20260808 04:45:24.966284 4964 test_util.cc:175] -----------------------------------------------
I20260808 04:45:24.966369 4964 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164296366965-4964-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (391 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 04:45:24.967514 4964 mini_oidc.cc:152] Starting JWKS server
I20260808 04:45:24.968529 4964 webserver.cc:460] Webserver started at https://127.0.0.1:44501/ using document root <none> and password file <none>
I20260808 04:45:24.968634 4964 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:45:24.968959 4964 webserver.cc:460] Webserver started at http://127.0.0.1:44421/ using document root <none> and password file <none>
2026-08-08T04:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:24Z Disabled control of system clock
I20260808 04:45:24.979015 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41497
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:35481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41497
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/testchainca.cert with env {}
W20260808 04:45:25.090492 13435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:25.090752 13435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:25.090792 13435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:25.094813 13435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:25.094867 13435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:25.094883 13435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:25.094930 13435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:25.094961 13435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:45:25.094987 13435 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:45:25.095021 13435 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44501/jwks.json
I20260808 04:45:25.100217 13435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:35481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:25.101271 13435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:25.102289 13435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:25.110977 13442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:25.111071 13435 server_base.cc:1034] running on GCE node
W20260808 04:45:25.111003 13448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:25.111003 13444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:25.111739 13435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:25.112859 13435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:25.114022 13435 hybrid_clock.cc:648] HybridClock initialized: now 1786164325113997 us; error 43 us; skew 500 ppm
I20260808 04:45:25.115881 13435 webserver.cc:460] Webserver started at http://127.4.217.62:44239/ using document root <none> and password file <none>
I20260808 04:45:25.116418 13435 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:25.116487 13435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:25.116653 13435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:25.118633 13435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "4fdda30efcb14fca95631db49b74dd66"
format_stamp: "Formatted at 2026-08-08 04:45:25 on dist-test-slave-2kcd"
server_key: "cf603e365019d996761f512f37a0ec6b"
server_key_iv: "91d31b122b7013a4c38c214bf2ee0b12"
server_key_version: "encryptionkey@0"
I20260808 04:45:25.119213 13435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "4fdda30efcb14fca95631db49b74dd66"
format_stamp: "Formatted at 2026-08-08 04:45:25 on dist-test-slave-2kcd"
server_key: "cf603e365019d996761f512f37a0ec6b"
server_key_iv: "91d31b122b7013a4c38c214bf2ee0b12"
server_key_version: "encryptionkey@0"
I20260808 04:45:25.122720 13435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:45:25.125000 13456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:25.126168 13435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:25.126317 13435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "4fdda30efcb14fca95631db49b74dd66"
format_stamp: "Formatted at 2026-08-08 04:45:25 on dist-test-slave-2kcd"
server_key: "cf603e365019d996761f512f37a0ec6b"
server_key_iv: "91d31b122b7013a4c38c214bf2ee0b12"
server_key_version: "encryptionkey@0"
I20260808 04:45:25.126438 13435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:25.152484 13435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:25.153151 13435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:25.153342 13435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:25.160352 13435 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:41497
I20260808 04:45:25.160389 13540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:41497 every 8 connection(s)
I20260808 04:45:25.161346 13435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:25.164413 13543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:25.164426 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 13435
I20260808 04:45:25.164556 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:25.164872 4964 external_mini_cluster.cc:1428] Setting key e54a141c7a33f3bc5c357b051d8ac641
I20260808 04:45:25.170769 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66: Bootstrap starting.
I20260808 04:45:25.172930 13543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:25.173695 13543 log.cc:824] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:25.173960 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:25.166223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.4.217.62:41497)
0808 04:45:25.166844 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:25.166863 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:25.166900 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:45:25.167591 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:25.167607 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:25.167968 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:25.168288 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:25.168743 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.169811 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.170719 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.171247 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.171437 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:25.172291 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:25.172296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:25.172307 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:25.172662 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:25.172714 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:25.172733 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:25.172884 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:25.173019 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 04:45:25.173258 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":301,"thread_start_us":137,"threads_started":1}
I20260808 04:45:25.175717 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66: No bootstrap required, opened a new log
I20260808 04:45:25.176654 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:45:25.179490 13543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdda30efcb14fca95631db49b74dd66" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41497 } }
I20260808 04:45:25.179742 13543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:25.179818 13543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdda30efcb14fca95631db49b74dd66, State: Initialized, Role: FOLLOWER
I20260808 04:45:25.180338 13543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdda30efcb14fca95631db49b74dd66" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41497 } }
I20260808 04:45:25.180522 13543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:25.180591 13543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:25.180673 13543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:25.181614 13543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdda30efcb14fca95631db49b74dd66" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41497 } }
I20260808 04:45:25.181900 13543 leader_election.cc:302] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fdda30efcb14fca95631db49b74dd66; no voters:
I20260808 04:45:25.182137 13543 leader_election.cc:288] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:25.182263 13549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:25.182483 13549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [term 1 LEADER]: Becoming Leader. State: Replica: 4fdda30efcb14fca95631db49b74dd66, State: Running, Role: LEADER
I20260808 04:45:25.182780 13549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdda30efcb14fca95631db49b74dd66" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41497 } }
I20260808 04:45:25.183319 13543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:25.184495 13551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fdda30efcb14fca95631db49b74dd66. Latest consensus state: current_term: 1 leader_uuid: "4fdda30efcb14fca95631db49b74dd66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdda30efcb14fca95631db49b74dd66" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41497 } } }
I20260808 04:45:25.184538 13550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fdda30efcb14fca95631db49b74dd66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdda30efcb14fca95631db49b74dd66" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41497 } } }
I20260808 04:45:25.184628 13551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:25.184645 13550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:25.185042 13562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:25.188042 13562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:25.193210 13562 catalog_manager.cc:1347] Generated new cluster ID: a770b8382b3f4461845193a6cdc1b153
I20260808 04:45:25.193318 13562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:25.221503 13562 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:25.222709 13562 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:25.235133 13562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fdda30efcb14fca95631db49b74dd66: Generated new TSK 0
I20260808 04:45:25.235829 13562 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 04:45:25.270736 13589 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 04:45:25.270830 13589 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 04:45:25.271504 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:25.252220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.4.217.62:41497)
0808 04:45:25.252358 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:25.252364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:25.252525 (+ 161us) server_negotiation.cc:407] Connection header received
0808 04:45:25.252665 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:25.252668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:25.252725 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:25.252871 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:45:25.253089 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.253913 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.254767 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.255132 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.255237 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:25.255510 (+ 273us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:45:25.271093 (+ 15583us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 04:45:25.271353 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 04:45:25.271584 13546 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 04:45:25.271616 13587 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:25.252109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41497 (local address 127.0.0.1:37308)
0808 04:45:25.252433 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:25.252451 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:25.252558 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:25.252872 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:25.252874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:25.252881 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:45:25.252977 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:25.252982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.254020 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.254022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:25.254655 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:25.254662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.255250 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.255253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:25.255318 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:25.255340 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:45:25.271289 (+ 15949us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 04:45:25.271438 (+ 149us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.217.62:41497: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260808 04:45:25.272357 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 13435
2026-08-08T04:45:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (353 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 04:45:25.321101 4964 mini_oidc.cc:152] Starting JWKS server
I20260808 04:45:25.322023 4964 webserver.cc:460] Webserver started at https://127.0.0.1:38685/ using document root <none> and password file <none>
I20260808 04:45:25.322173 4964 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:45:25.322448 4964 webserver.cc:460] Webserver started at http://127.0.0.1:41645/ using document root <none> and password file <none>
2026-08-08T04:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:25Z Disabled control of system clock
I20260808 04:45:25.333056 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:36631
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:36631
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38685/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 04:45:25.448215 13596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:25.448482 13596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:25.448530 13596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:25.452670 13596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:25.452730 13596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:25.452746 13596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:25.452762 13596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:25.452792 13596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:45:25.452847 13596 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:45:25.452883 13596 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38685/jwks.json
I20260808 04:45:25.457655 13596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:36631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38685/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:25.458841 13596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:25.460357 13596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:25.469085 13604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:25.469110 13608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:25.469141 13605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:25.469563 13596 server_base.cc:1034] running on GCE node
I20260808 04:45:25.470187 13596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:25.471351 13596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:25.472565 13596 hybrid_clock.cc:648] HybridClock initialized: now 1786164325472537 us; error 51 us; skew 500 ppm
I20260808 04:45:25.474740 13596 webserver.cc:460] Webserver started at http://127.4.217.62:45321/ using document root <none> and password file <none>
I20260808 04:45:25.475370 13596 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:25.475472 13596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:25.475706 13596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:25.478032 13596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "a642e4c7da814805af7c7620d2773c42"
format_stamp: "Formatted at 2026-08-08 04:45:25 on dist-test-slave-2kcd"
server_key: "2e49d9b357aaf8612c9125b122d0aaf5"
server_key_iv: "27952fd264d3a8e61446c89023b00333"
server_key_version: "encryptionkey@0"
I20260808 04:45:25.478665 13596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "a642e4c7da814805af7c7620d2773c42"
format_stamp: "Formatted at 2026-08-08 04:45:25 on dist-test-slave-2kcd"
server_key: "2e49d9b357aaf8612c9125b122d0aaf5"
server_key_iv: "27952fd264d3a8e61446c89023b00333"
server_key_version: "encryptionkey@0"
I20260808 04:45:25.482647 13596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:45:25.485280 13619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:25.486564 13596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:25.486721 13596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "a642e4c7da814805af7c7620d2773c42"
format_stamp: "Formatted at 2026-08-08 04:45:25 on dist-test-slave-2kcd"
server_key: "2e49d9b357aaf8612c9125b122d0aaf5"
server_key_iv: "27952fd264d3a8e61446c89023b00333"
server_key_version: "encryptionkey@0"
I20260808 04:45:25.486846 13596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:25.501174 13596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:25.501860 13596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:25.502063 13596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:25.508872 13596 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:36631
I20260808 04:45:25.508898 13705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:36631 every 8 connection(s)
I20260808 04:45:25.509811 13596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:25.512985 13707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:25.518384 13707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42: Bootstrap starting.
I20260808 04:45:25.519228 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 13596
I20260808 04:45:25.519353 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:25.519662 4964 external_mini_cluster.cc:1428] Setting key 0463f3997d80d24b06bb0f9b08fa80df
I20260808 04:45:25.521423 13707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:25.522167 13707 log.cc:824] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:25.524047 13707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42: No bootstrap required, opened a new log
I20260808 04:45:25.527649 13707 raft_consensus.cc:348] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a642e4c7da814805af7c7620d2773c42" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36631 } }
I20260808 04:45:25.527889 13707 raft_consensus.cc:374] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:25.527951 13707 raft_consensus.cc:729] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a642e4c7da814805af7c7620d2773c42, State: Initialized, Role: FOLLOWER
I20260808 04:45:25.528427 13707 consensus_queue.cc:260] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a642e4c7da814805af7c7620d2773c42" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36631 } }
I20260808 04:45:25.528586 13707 raft_consensus.cc:388] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:25.528674 13707 raft_consensus.cc:482] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:25.528820 13707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:25.528980 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:25.520856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.4.217.62:36631)
0808 04:45:25.521493 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:25.521507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:25.521590 (+ 83us) server_negotiation.cc:407] Connection header received
0808 04:45:25.522194 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:25.522210 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:25.522739 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:25.523057 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:25.523483 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.524472 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.525454 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.525929 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.526112 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:25.527033 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:25.527039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:25.527053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:25.527423 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:25.527478 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:25.527498 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:25.527622 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:25.527771 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 04:45:25.528042 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":141,"threads_started":1}
I20260808 04:45:25.529772 13707 raft_consensus.cc:504] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a642e4c7da814805af7c7620d2773c42" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36631 } }
I20260808 04:45:25.530125 13707 leader_election.cc:302] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a642e4c7da814805af7c7620d2773c42; no voters:
I20260808 04:45:25.530434 13707 leader_election.cc:288] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:25.530536 13714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:25.531073 13714 raft_consensus.cc:686] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [term 1 LEADER]: Becoming Leader. State: Replica: a642e4c7da814805af7c7620d2773c42, State: Running, Role: LEADER
I20260808 04:45:25.531389 13714 consensus_queue.cc:237] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a642e4c7da814805af7c7620d2773c42" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36631 } }
I20260808 04:45:25.531796 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:45:25.531795 13707 sys_catalog.cc:564] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:25.533440 13716 sys_catalog.cc:455] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a642e4c7da814805af7c7620d2773c42. Latest consensus state: current_term: 1 leader_uuid: "a642e4c7da814805af7c7620d2773c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a642e4c7da814805af7c7620d2773c42" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36631 } } }
I20260808 04:45:25.533435 13715 sys_catalog.cc:455] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a642e4c7da814805af7c7620d2773c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a642e4c7da814805af7c7620d2773c42" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36631 } } }
I20260808 04:45:25.533604 13715 sys_catalog.cc:458] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:25.533604 13716 sys_catalog.cc:458] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:25.534183 13735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:25.537117 13735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:25.542586 13735 catalog_manager.cc:1347] Generated new cluster ID: 7aa068963237442f9ede707902c80cac
I20260808 04:45:25.542654 13735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:25.573493 13735 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:25.574421 13735 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:25.580482 13735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a642e4c7da814805af7c7620d2773c42: Generated new TSK 0
I20260808 04:45:25.581276 13735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:45:25.595072 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:25.588968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.4.217.62:36631)
0808 04:45:25.589101 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:25.589106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:25.589269 (+ 163us) server_negotiation.cc:407] Connection header received
0808 04:45:25.589395 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:25.589398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:25.589453 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:25.589568 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:45:25.589775 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.590648 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.591720 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.592088 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.592191 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:25.592483 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:45:25.594688 (+ 2205us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 04:45:25.594928 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:45:25.595137 13711 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 04:45:25.596813 13754 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:25.588909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:36631 (local address 127.0.0.1:40230)
0808 04:45:25.589181 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:25.589194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:25.589290 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:25.589571 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:25.589573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:25.589580 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:45:25.589672 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:25.589676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.590789 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.590792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:25.591563 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:25.591573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.592223 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.592226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:25.592291 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:25.592314 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:45:25.594889 (+ 2575us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 04:45:25.596609 (+ 1720us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.217.62:36631: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1519,"thread_start_us":109,"threads_started":1}
I20260808 04:45:25.597368 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 13596
E20260808 04:45:25.614991 13756 webserver.cc:533] Webserver: sslize failed:
E20260808 04:45:25.615077 13756 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T04:45:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T04:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:25Z Disabled control of system clock
I20260808 04:45:25.658447 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:43699
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:43699
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/testchainca.cert with env {}
W20260808 04:45:25.774400 13760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:25.774682 13760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:25.774734 13760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:25.778694 13760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:25.778752 13760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:25.778771 13760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:25.778787 13760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:25.778802 13760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:45:25.778818 13760 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:45:25.778846 13760 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 04:45:25.784214 13760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:43699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:43699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:25.785313 13760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:25.786320 13760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:25.795267 13770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:25.795262 13779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:25.795262 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:25.795410 13760 server_base.cc:1034] running on GCE node
I20260808 04:45:25.796168 13760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:25.797398 13760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:25.798594 13760 hybrid_clock.cc:648] HybridClock initialized: now 1786164325798581 us; error 25 us; skew 500 ppm
I20260808 04:45:25.800434 13760 webserver.cc:460] Webserver started at http://127.4.217.62:42235/ using document root <none> and password file <none>
I20260808 04:45:25.801117 13760 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:25.801215 13760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:25.801452 13760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:25.803543 13760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "e334dbdf6d454767b7fd8448025ade28"
format_stamp: "Formatted at 2026-08-08 04:45:25 on dist-test-slave-2kcd"
server_key: "b267b182f5253a580b69190cce06ef49"
server_key_iv: "18c36d9373ef679ea24de03e6f3c7446"
server_key_version: "encryptionkey@0"
I20260808 04:45:25.804133 13760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "e334dbdf6d454767b7fd8448025ade28"
format_stamp: "Formatted at 2026-08-08 04:45:25 on dist-test-slave-2kcd"
server_key: "b267b182f5253a580b69190cce06ef49"
server_key_iv: "18c36d9373ef679ea24de03e6f3c7446"
server_key_version: "encryptionkey@0"
I20260808 04:45:25.807837 13760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:45:25.810276 13788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:25.811501 13760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:45:25.811657 13760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "e334dbdf6d454767b7fd8448025ade28"
format_stamp: "Formatted at 2026-08-08 04:45:25 on dist-test-slave-2kcd"
server_key: "b267b182f5253a580b69190cce06ef49"
server_key_iv: "18c36d9373ef679ea24de03e6f3c7446"
server_key_version: "encryptionkey@0"
I20260808 04:45:25.811776 13760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:25.841437 13760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:25.842177 13760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:25.842391 13760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:25.849565 13760 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:43699
I20260808 04:45:25.849570 13872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:43699 every 8 connection(s)
I20260808 04:45:25.850662 13760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:25.853901 13874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:25.854393 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 13760
I20260808 04:45:25.854522 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:25.854864 4964 external_mini_cluster.cc:1428] Setting key 984d9ba8df0f107221433326e42cc563
I20260808 04:45:25.860905 13874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28: Bootstrap starting.
I20260808 04:45:25.863574 13874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:25.864149 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:25.856195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.4.217.62:43699)
0808 04:45:25.856896 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:25.856912 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:25.856949 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:45:25.857622 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:25.857640 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:25.858018 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:25.858367 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:25.858791 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.859746 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.860649 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.861228 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.861399 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:25.862354 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:25.862360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:25.862372 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:25.862756 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:25.862809 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:25.862831 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:25.862948 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:25.863115 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:45:25.863367 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260808 04:45:25.864444 13874 log.cc:824] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:25.866652 13874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28: No bootstrap required, opened a new log
I20260808 04:45:25.867003 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:45:25.870635 13874 raft_consensus.cc:348] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e334dbdf6d454767b7fd8448025ade28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43699 } }
I20260808 04:45:25.870890 13874 raft_consensus.cc:374] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:25.870978 13874 raft_consensus.cc:729] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e334dbdf6d454767b7fd8448025ade28, State: Initialized, Role: FOLLOWER
I20260808 04:45:25.871517 13874 consensus_queue.cc:260] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e334dbdf6d454767b7fd8448025ade28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43699 } }
I20260808 04:45:25.871695 13874 raft_consensus.cc:388] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:25.871769 13874 raft_consensus.cc:482] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:25.871873 13874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:25.872895 13874 raft_consensus.cc:504] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e334dbdf6d454767b7fd8448025ade28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43699 } }
I20260808 04:45:25.873263 13874 leader_election.cc:302] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e334dbdf6d454767b7fd8448025ade28; no voters:
I20260808 04:45:25.873608 13874 leader_election.cc:288] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:25.873737 13881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:25.873994 13881 raft_consensus.cc:686] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [term 1 LEADER]: Becoming Leader. State: Replica: e334dbdf6d454767b7fd8448025ade28, State: Running, Role: LEADER
I20260808 04:45:25.874297 13881 consensus_queue.cc:237] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e334dbdf6d454767b7fd8448025ade28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43699 } }
I20260808 04:45:25.874958 13874 sys_catalog.cc:564] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:25.876142 13884 sys_catalog.cc:455] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e334dbdf6d454767b7fd8448025ade28. Latest consensus state: current_term: 1 leader_uuid: "e334dbdf6d454767b7fd8448025ade28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e334dbdf6d454767b7fd8448025ade28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43699 } } }
I20260808 04:45:25.876307 13884 sys_catalog.cc:458] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:25.876593 13883 sys_catalog.cc:455] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e334dbdf6d454767b7fd8448025ade28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e334dbdf6d454767b7fd8448025ade28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43699 } } }
I20260808 04:45:25.876700 13883 sys_catalog.cc:458] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:25.877166 13897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:25.880474 13897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:25.885694 13897 catalog_manager.cc:1347] Generated new cluster ID: 9e38defbd49242cd937804e3e1c8b59c
I20260808 04:45:25.885758 13897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:25.901986 13897 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:25.902925 13897 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:25.908622 13897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e334dbdf6d454767b7fd8448025ade28: Generated new TSK 0
I20260808 04:45:25.909299 13897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:45:26.400628 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:25.917143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.4.217.62:43699)
0808 04:45:25.917266 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:25.917270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:25.917461 (+ 191us) server_negotiation.cc:407] Connection header received
0808 04:45:25.917668 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:25.917671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:25.917719 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:25.917828 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:45:25.918086 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.918617 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.919496 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.919722 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.919827 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:25.920171 (+ 344us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:45:26.400098 (+479927us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 04:45:26.400453 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58242: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 04:45:26.400734 13877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58242: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 04:45:26.400802 13927 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:25.917082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:43699 (local address 127.0.0.1:58242)
0808 04:45:25.917371 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:25.917383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:25.917514 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:25.917840 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:25.917843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:25.917851 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:45:25.917963 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:25.917975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.918735 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.918739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:25.919364 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:25.919371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:25.919863 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:25.919870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:25.919966 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:25.919988 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:45:26.400355 (+480367us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 04:45:26.400576 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.217.62:43699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
I20260808 04:45:26.401368 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 13760
2026-08-08T04:45:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (781 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (130 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:26 dist-test-slave-2kcd krb5kdc[13948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:26 dist-test-slave-2kcd krb5kdc[13948](info): set up 2 sockets
Aug 08 04:45:26 dist-test-slave-2kcd krb5kdc[13948](info): commencing operation
krb5kdc: starting...
W20260808 04:45:27.926901 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:28 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:28.051080 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35355
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:28.163252 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:28.163519 13968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:28.163559 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:28.167804 13968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:28.167876 13968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:28.167897 13968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:28.167977 13968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:28.168020 13968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:28.173517 13968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:28.174574 13968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:28.175611 13968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:28.184567 13975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:28.184652 13968 server_base.cc:1034] running on GCE node
W20260808 04:45:28.184551 13980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:28.184552 13977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:28.185472 13968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:28.186556 13968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:28.187752 13968 hybrid_clock.cc:648] HybridClock initialized: now 1786164328187734 us; error 30 us; skew 500 ppm
Aug 08 04:45:28 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:28.193733 13968 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:28.194696 13968 webserver.cc:460] Webserver started at http://127.4.217.62:36735/ using document root <none> and password file <none>
I20260808 04:45:28.195237 13968 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:28.195304 13968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:28.195479 13968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:28.197611 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "cfb7932100aa4e3283a5ffed9c394884"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "08a8be29c31130ead6549ff65fc79208"
server_key_iv: "9aa73c1a4aefcd62ff88121ef40982b5"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.198137 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "cfb7932100aa4e3283a5ffed9c394884"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "08a8be29c31130ead6549ff65fc79208"
server_key_iv: "9aa73c1a4aefcd62ff88121ef40982b5"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.202041 13968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:45:28.204411 13990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.205691 13968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:28.205828 13968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "cfb7932100aa4e3283a5ffed9c394884"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "08a8be29c31130ead6549ff65fc79208"
server_key_iv: "9aa73c1a4aefcd62ff88121ef40982b5"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.205931 13968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:28.221663 13968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:28.224339 13968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:28.224555 13968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:28.232435 13968 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:35355
I20260808 04:45:28.232465 14072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:35355 every 8 connection(s)
I20260808 04:45:28.233412 13968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:28.236160 14074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:28.237143 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 13968
I20260808 04:45:28.237286 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:28.237582 4964 external_mini_cluster.cc:1428] Setting key 22829403e93b1ac0fc7eb5dc75edb822
I20260808 04:45:28.242491 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884: Bootstrap starting.
I20260808 04:45:28.245261 14074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:28.246239 14074 log.cc:824] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:45:28 dist-test-slave-2kcd krb5kdc[13948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:28.248957 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884: No bootstrap required, opened a new log
I20260808 04:45:28.254127 14074 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb7932100aa4e3283a5ffed9c394884" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35355 } }
I20260808 04:45:28.254487 14074 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:28.254570 14074 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb7932100aa4e3283a5ffed9c394884, State: Initialized, Role: FOLLOWER
I20260808 04:45:28.255139 14074 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb7932100aa4e3283a5ffed9c394884" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35355 } }
I20260808 04:45:28.255313 14074 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:28.255389 14074 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:28.255496 14074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:28.256733 14074 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb7932100aa4e3283a5ffed9c394884" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35355 } }
I20260808 04:45:28.257174 14074 leader_election.cc:302] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfb7932100aa4e3283a5ffed9c394884; no voters:
I20260808 04:45:28.257535 14074 leader_election.cc:288] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:28.257927 14081 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:28.258091 14081 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb7932100aa4e3283a5ffed9c394884" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35355 } }
I20260808 04:45:28.258312 14082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:28.258530 14082 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [term 1 LEADER]: Becoming Leader. State: Replica: cfb7932100aa4e3283a5ffed9c394884, State: Running, Role: LEADER
I20260808 04:45:28.258733 14081 leader_election.cc:302] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfb7932100aa4e3283a5ffed9c394884; no voters:
I20260808 04:45:28.258827 14082 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb7932100aa4e3283a5ffed9c394884" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35355 } }
I20260808 04:45:28.259020 14081 leader_election.cc:288] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260808 04:45:28.259014 14074 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:28.260399 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfb7932100aa4e3283a5ffed9c394884. Latest consensus state: current_term: 1 leader_uuid: "cfb7932100aa4e3283a5ffed9c394884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb7932100aa4e3283a5ffed9c394884" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35355 } } }
I20260808 04:45:28.260566 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:28.261058 14093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:28.261606 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:28.238937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.4.217.62:35355)
0808 04:45:28.239349 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:28.239364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:28.239398 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:45:28.240006 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:28.240022 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:28.240320 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:28.240601 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:28.241567 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.242491 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.243675 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.244120 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.244245 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:28.247233 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:28.247240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:28.247256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:28.247285 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:28.249371 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:28.249948 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:28.249954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:28.249962 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:28.250103 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:28.250454 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:28.250458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:28.250487 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:28.251327 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:28.251679 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:28.260121 (+ 8442us) server_negotiation.cc:299] Negotiation successful
0808 04:45:28.260524 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":131,"threads_started":1}
I20260808 04:45:28.262331 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfb7932100aa4e3283a5ffed9c394884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb7932100aa4e3283a5ffed9c394884" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35355 } } }
I20260808 04:45:28.262430 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:28.266556 14093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:28.273512 14093 catalog_manager.cc:1347] Generated new cluster ID: 8a60623b6af646b284df9532aa4e447a
I20260808 04:45:28.273579 14093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:28.298806 14093 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:28.300163 14093 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:28.306434 14093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfb7932100aa4e3283a5ffed9c394884: Generated new TSK 0
I20260808 04:45:28.307142 14093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:45:28.343253 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35355
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:28.454028 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:28.454295 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:28.454341 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:28.458597 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:28.458660 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:28.458760 14126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:28.464253 14126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:28.465374 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:28.466432 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:28.476480 14131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:28.476464 14132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:28.476733 14126 server_base.cc:1034] running on GCE node
W20260808 04:45:28.476464 14135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:28.477342 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:28.477973 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:28.479157 14126 hybrid_clock.cc:648] HybridClock initialized: now 1786164328479130 us; error 29 us; skew 500 ppm
Aug 08 04:45:28 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:28.481962 14126 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:45:28.482829 14126 webserver.cc:460] Webserver started at http://127.4.217.1:39049/ using document root <none> and password file <none>
I20260808 04:45:28.483407 14126 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:28.483516 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:28.483752 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:28.485893 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "3b855cef311c463ead60fc20991b6d72"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "1618d78e7b96604fc17af4baea96765f"
server_key_iv: "619203b46af8742337b7a917004a5078"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.486505 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "3b855cef311c463ead60fc20991b6d72"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "1618d78e7b96604fc17af4baea96765f"
server_key_iv: "619203b46af8742337b7a917004a5078"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.490236 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:28.492597 14143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.493821 14126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:28.493961 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "3b855cef311c463ead60fc20991b6d72"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "1618d78e7b96604fc17af4baea96765f"
server_key_iv: "619203b46af8742337b7a917004a5078"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.494074 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:28.509155 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:28.512069 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:28.512219 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:28.512737 14126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:28.513592 14126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:28.513639 14126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.513679 14126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:28.513695 14126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.522745 14126 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:33231
I20260808 04:45:28.522775 14325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:33231 every 8 connection(s)
I20260808 04:45:28.523638 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:28.529217 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 14126
I20260808 04:45:28.529379 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:28.529676 4964 external_mini_cluster.cc:1428] Setting key 3c32fda451bc4a65eb50de90c0bc5c75
Aug 08 04:45:28 dist-test-slave-2kcd krb5kdc[13948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:28.539000 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:28.525782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:40903 (local address 127.4.217.62:35355)
0808 04:45:28.526003 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:28.526007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:28.526785 (+ 778us) server_negotiation.cc:407] Connection header received
0808 04:45:28.528293 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:28.528296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:28.528359 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:28.528443 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:28.529930 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.530483 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.531636 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.531890 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.532008 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:28.534823 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:28.534830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:28.534836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:28.534868 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:28.536598 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:28.537208 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:28.537214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:28.537217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:28.537317 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:28.537703 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:28.537710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:28.537715 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:28.537991 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:28.538125 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:28.538649 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 04:45:28.538801 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:45:28.539724 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:28.526173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35355 (local address 127.4.217.1:40903)
0808 04:45:28.526634 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:28.526678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:28.528063 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:28.528645 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:28.528653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:28.529085 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:28.529697 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:28.529712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.530620 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.530623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:28.531501 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:28.531511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.532052 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.532056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:28.532225 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:28.532932 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:28.532948 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:28.534646 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:28.536740 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:28.536750 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:28.536766 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:28.537093 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:28.537426 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:28.537429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:28.537430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:28.537571 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:28.538126 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:28.538132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:28.538376 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 04:45:28.538532 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 04:45:28.538784 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260808 04:45:28.541133 14327 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35355
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
I20260808 04:45:28.541420 14327 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:28.541946 14327 heartbeater.cc:511] Master 127.4.217.62:35355 requested a full tablet report, sending...
I20260808 04:45:28.543614 14015 ts_manager.cc:194] Registered new tserver with Master: 3b855cef311c463ead60fc20991b6d72 (127.4.217.1:33231)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:45:28.545024 14015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:40903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:45:28.560390 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35355
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:28.675098 14334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:28.675376 14334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:28.675453 14334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:28.679450 14334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:28.679564 14334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:28.679672 14334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:28.685043 14334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:28.686053 14334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:28.687086 14334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:28.697170 14342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:28.697304 14334 server_base.cc:1034] running on GCE node
W20260808 04:45:28.697216 14348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:28.697187 14343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:28.697834 14334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:28.698479 14334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:28.699654 14334 hybrid_clock.cc:648] HybridClock initialized: now 1786164328699622 us; error 44 us; skew 500 ppm
Aug 08 04:45:28 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:28.703088 14334 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:45:28.704207 14334 webserver.cc:460] Webserver started at http://127.4.217.2:46533/ using document root <none> and password file <none>
I20260808 04:45:28.704870 14334 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:28.704977 14334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:28.705202 14334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:28.707348 14334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "f60c60cf7fde4ad38e9125b7847871ed"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "76d63211fe36f59fd72a85130d46fb25"
server_key_iv: "3c89c6496b167c0365286a5b5ff11433"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.707959 14334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "f60c60cf7fde4ad38e9125b7847871ed"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "76d63211fe36f59fd72a85130d46fb25"
server_key_iv: "3c89c6496b167c0365286a5b5ff11433"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.711949 14334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:45:28.714543 14361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.715758 14334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:45:28.715901 14334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "f60c60cf7fde4ad38e9125b7847871ed"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "76d63211fe36f59fd72a85130d46fb25"
server_key_iv: "3c89c6496b167c0365286a5b5ff11433"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.716017 14334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:28.731398 14334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:28.734377 14334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:28.734586 14334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:28.735175 14334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:28.736027 14334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:28.736107 14334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.736179 14334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:28.736232 14334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.745368 14334 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:37591
I20260808 04:45:28.745395 14547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:37591 every 8 connection(s)
I20260808 04:45:28.746229 14334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:28.746881 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 14334
I20260808 04:45:28.747010 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:28.747327 4964 external_mini_cluster.cc:1428] Setting key 5cfc183bd41cdfb5fd00af39276cd10f
Aug 08 04:45:28 dist-test-slave-2kcd krb5kdc[13948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
I20260808 04:45:28.763037 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:28.748246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:37615 (local address 127.4.217.62:35355)
0808 04:45:28.748394 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:28.748399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:28.749560 (+ 1161us) server_negotiation.cc:407] Connection header received
0808 04:45:28.751623 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:28.751628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:28.751690 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:28.751787 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:28.753132 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.753654 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.755526 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.755734 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.755814 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:28.758680 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:28.758685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:28.758691 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:28.758721 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:28.760431 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:28.761110 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:28.761113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:28.761114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:28.761191 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:28.761505 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:28.761507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:28.761508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:28.761706 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:28.761809 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:28.762444 (+ 635us) server_negotiation.cc:299] Negotiation successful
0808 04:45:28.762588 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:45:28.763602 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:28.748768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35355 (local address 127.4.217.2:37615)
0808 04:45:28.749382 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:28.749440 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:28.751393 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:28.751972 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:28.751978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:28.752319 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:28.752959 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:28.752983 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.753787 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.753794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:28.755391 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:28.755407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.755909 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.755915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:28.756156 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:28.756774 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:28.756830 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:28.758495 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:28.760555 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:28.760561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:28.760571 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:28.761005 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:28.761268 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:28.761270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:28.761271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:28.761406 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:28.761813 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:28.761822 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:28.762175 (+ 353us) client_negotiation.cc:769] Sending connection context
0808 04:45:28.762332 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 04:45:28.762614 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":152,"threads_started":1}
I20260808 04:45:28.765077 14548 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35355
I20260808 04:45:28.765333 14548 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:28.765813 14548 heartbeater.cc:511] Master 127.4.217.62:35355 requested a full tablet report, sending...
I20260808 04:45:28.766925 14015 ts_manager.cc:194] Registered new tserver with Master: f60c60cf7fde4ad38e9125b7847871ed (127.4.217.2:37591)
I20260808 04:45:28.767815 14015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:37615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:45:28.779525 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35355
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:28.891829 14555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:28.892064 14555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:28.892105 14555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:28.896026 14555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:28.896078 14555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:28.896133 14555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:28.901225 14555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:28.902129 14555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:28.903075 14555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:28.912573 14561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:28.912662 14555 server_base.cc:1034] running on GCE node
W20260808 04:45:28.912575 14562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:28.912616 14565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:28.913221 14555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:28.914038 14555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:28.915241 14555 hybrid_clock.cc:648] HybridClock initialized: now 1786164328915215 us; error 40 us; skew 500 ppm
Aug 08 04:45:28 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:28.917991 14555 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:45:28.918882 14555 webserver.cc:460] Webserver started at http://127.4.217.3:37359/ using document root <none> and password file <none>
I20260808 04:45:28.919450 14555 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:28.919512 14555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:28.919723 14555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:28.921869 14555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "ab41ae5ac963407baba899103dffcf2e"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "4c1a6b41d40b3dc74f9bd0600a26c409"
server_key_iv: "462654f57e944e4403e53fd52e7e8b66"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.922430 14555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "ab41ae5ac963407baba899103dffcf2e"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "4c1a6b41d40b3dc74f9bd0600a26c409"
server_key_iv: "462654f57e944e4403e53fd52e7e8b66"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.926100 14555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:28.928468 14575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.929729 14555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:28.929853 14555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "ab41ae5ac963407baba899103dffcf2e"
format_stamp: "Formatted at 2026-08-08 04:45:28 on dist-test-slave-2kcd"
server_key: "4c1a6b41d40b3dc74f9bd0600a26c409"
server_key_iv: "462654f57e944e4403e53fd52e7e8b66"
server_key_version: "encryptionkey@0"
I20260808 04:45:28.929957 14555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:28.954238 14555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:28.957376 14555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:28.957602 14555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:28.958249 14555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:28.959102 14555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:28.959157 14555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.959229 14555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:28.959270 14555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:28.968703 14555 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:38027
I20260808 04:45:28.968735 14751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:38027 every 8 connection(s)
I20260808 04:45:28.969650 14555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:28.975747 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 14555
I20260808 04:45:28.975870 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:28.976157 4964 external_mini_cluster.cc:1428] Setting key 6630416bfe2117ed65b1fa4a200cee23
Aug 08 04:45:28 dist-test-slave-2kcd krb5kdc[13948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:28.984063 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:28.971630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:46961 (local address 127.4.217.62:35355)
0808 04:45:28.971770 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:28.971774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:28.972630 (+ 856us) server_negotiation.cc:407] Connection header received
0808 04:45:28.974109 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:28.974113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:28.974174 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:28.974282 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:28.975434 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.976237 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.977952 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.978294 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.978410 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:28.980849 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:28.980852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:28.980854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:28.980878 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:28.982317 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:28.982792 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:28.982794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:28.982795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:28.982869 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:28.983181 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:28.983183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:28.983184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:28.983344 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:28.983420 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:28.983853 (+ 433us) server_negotiation.cc:299] Negotiation successful
0808 04:45:28.983946 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:45:28.984905 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:28.972019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35355 (local address 127.4.217.3:46961)
0808 04:45:28.972478 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:28.972518 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:28.973894 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:28.974464 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:28.974470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:28.974820 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:28.975284 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:28.975297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.976385 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.976388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:28.977824 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:28.977839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:28.978485 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:28.978490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:28.978678 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:28.979204 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:28.979221 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:28.980649 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:28.982430 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:28.982435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:28.982445 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:28.982703 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:28.982965 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:28.982967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:28.982969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:28.983098 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:28.983438 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:28.983442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:28.983652 (+ 210us) client_negotiation.cc:769] Sending connection context
0808 04:45:28.983801 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:45:28.984060 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":118,"threads_started":1}
I20260808 04:45:28.986014 14753 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35355
I20260808 04:45:28.986212 14753 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:28.986634 14753 heartbeater.cc:511] Master 127.4.217.62:35355 requested a full tablet report, sending...
I20260808 04:45:28.987569 14015 ts_manager.cc:194] Registered new tserver with Master: ab41ae5ac963407baba899103dffcf2e (127.4.217.3:38027)
I20260808 04:45:28.988095 14015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:46961
I20260808 04:45:28.990166 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/test-admin.keytab.
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:29.009402 4964 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 04:45:29.012225 14766 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:29.010780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.4.217.1:33231)
0808 04:45:29.011075 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:29.011090 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:29.011113 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:45:29.011206 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:29.011211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:29.011391 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:29.011505 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:29.012051 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60952: BlockingRecv error: recv got EOF from 127.0.0.1:60952 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
W20260808 04:45:29.012297 4964 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:29.013599 4964 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:29.014592 4964 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:45:29.023990 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:29.015408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.4.217.1:33231)
0808 04:45:29.015541 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:29.015545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:29.015661 (+ 116us) server_negotiation.cc:407] Connection header received
0808 04:45:29.015720 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:29.015723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:29.015758 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:29.015822 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:29.016316 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:29.017189 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:29.017975 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:29.018312 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:29.018397 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:29.020634 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:29.020643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:29.020655 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:29.020676 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:29.022375 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:29.022753 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:29.022759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:29.022760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:29.022836 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:29.023101 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:29.023103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:29.023104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:29.023320 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:29.023463 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:29.023633 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 04:45:29.023797 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 04:45:29.027729 14766 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:29.026707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.4.217.1:33231)
0808 04:45:29.026835 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:29.026839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:29.026946 (+ 107us) server_negotiation.cc:407] Connection header received
0808 04:45:29.027103 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:29.027106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:29.027145 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:29.027236 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:29.027638 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60968: BlockingRecv error: recv got EOF from 127.0.0.1:60968 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:45:29.027858 4964 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:29.028828 4964 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:29.029740 4964 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:45:29.038296 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:29.030496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.4.217.1:33231)
0808 04:45:29.030626 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:29.030630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:29.030748 (+ 118us) server_negotiation.cc:407] Connection header received
0808 04:45:29.030883 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:29.030886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:29.030922 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:29.031011 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:29.031501 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:29.032012 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:29.032838 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:29.033091 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:29.033203 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:29.035364 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:29.035367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:29.035370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:29.035394 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:29.036697 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:29.037141 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:29.037145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:29.037146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:29.037230 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:29.037503 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:29.037506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:29.037508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:29.037679 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:29.037809 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:29.038043 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 04:45:29.038148 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 04:45:29.040964 14766 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:29.039923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.4.217.1:33231)
0808 04:45:29.040058 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:29.040062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:29.040187 (+ 125us) server_negotiation.cc:407] Connection header received
0808 04:45:29.040317 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:29.040320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:29.040356 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:29.040444 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:29.040859 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60990: BlockingRecv error: recv got EOF from 127.0.0.1:60990 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 04:45:29.041206 4964 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164296366965-4964-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:29.042131 4964 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:29.042966 4964 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[13948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:45:29.050729 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:29.043689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.4.217.1:33231)
0808 04:45:29.043817 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:29.043820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:29.043835 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:29.043940 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:29.043942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:29.043973 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:29.044035 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:29.044455 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:29.045010 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:29.045716 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:29.045937 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:29.046014 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:29.047947 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:29.047950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:29.047952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:29.047972 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:29.049358 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:29.049724 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:29.049726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:29.049727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:29.049801 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:29.050045 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:29.050047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:29.050048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:29.050183 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:29.050273 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:29.050462 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 04:45:29.050578 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:45:29.051673 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 14126
I20260808 04:45:29.076581 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 14334
I20260808 04:45:29.087898 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 14555
I20260808 04:45:29.113766 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 13968
2026-08-08T04:45:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2450 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[14792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[14792](info): set up 2 sockets
Aug 08 04:45:29 dist-test-slave-2kcd krb5kdc[14792](info): commencing operation
krb5kdc: starting...
W20260808 04:45:30.386165 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:30 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.62.keytab.
Entry for principal oryx/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.62.keytab.
I20260808 04:45:30.512768 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33995
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.62.keytab
--principal=oryx/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:33995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:30.624534 14811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:30.624769 14811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:30.624868 14811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:30.628772 14811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:30.628921 14811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:30.628943 14811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:30.628959 14811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:30.628979 14811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:30.634004 14811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:33995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.62.keytab
--principal=oryx/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:30.634881 14811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:30.635764 14811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:30.644110 14826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:30.644138 14822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:30.644207 14811 server_base.cc:1034] running on GCE node
W20260808 04:45:30.644148 14823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:30.644954 14811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:30.645967 14811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:30.647159 14811 hybrid_clock.cc:648] HybridClock initialized: now 1786164330647129 us; error 43 us; skew 500 ppm
Aug 08 04:45:30 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164330, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:30.649909 14811 init.cc:377] Logged in from keytab as oryx/127.4.217.62@KRBTEST.COM (short username oryx)
I20260808 04:45:30.650771 14811 webserver.cc:460] Webserver started at http://127.4.217.62:35219/ using document root <none> and password file <none>
I20260808 04:45:30.651315 14811 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:30.651400 14811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:30.651623 14811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:30.653614 14811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "27cbf01bd965461fba0d841fd3a2ef81"
format_stamp: "Formatted at 2026-08-08 04:45:30 on dist-test-slave-2kcd"
server_key: "856bd8f502f08928b9c4a0ac0bb81194"
server_key_iv: "63946465dcd0a117c0022c45331b3bb1"
server_key_version: "encryptionkey@0"
I20260808 04:45:30.654174 14811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "27cbf01bd965461fba0d841fd3a2ef81"
format_stamp: "Formatted at 2026-08-08 04:45:30 on dist-test-slave-2kcd"
server_key: "856bd8f502f08928b9c4a0ac0bb81194"
server_key_iv: "63946465dcd0a117c0022c45331b3bb1"
server_key_version: "encryptionkey@0"
I20260808 04:45:30.657739 14811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:30.659960 14836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:30.661118 14811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:30.661252 14811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "27cbf01bd965461fba0d841fd3a2ef81"
format_stamp: "Formatted at 2026-08-08 04:45:30 on dist-test-slave-2kcd"
server_key: "856bd8f502f08928b9c4a0ac0bb81194"
server_key_iv: "63946465dcd0a117c0022c45331b3bb1"
server_key_version: "encryptionkey@0"
I20260808 04:45:30.661356 14811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:30.679203 14811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:30.681999 14811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:30.682216 14811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:30.689630 14811 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:33995
I20260808 04:45:30.689941 14919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:33995 every 8 connection(s)
I20260808 04:45:30.690696 14811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:30.693580 14920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:30.698407 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 14811
I20260808 04:45:30.698523 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:30.698807 4964 external_mini_cluster.cc:1428] Setting key af41f2df28daa30293ee8a8621923bbe
I20260808 04:45:30.699069 14920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: Bootstrap starting.
I20260808 04:45:30.701629 14920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:30.702360 14920 log.cc:824] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:30.704223 14920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: No bootstrap required, opened a new log
Aug 08 04:45:30 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:30.708027 14920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } }
I20260808 04:45:30.708315 14920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:30.708396 14920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cbf01bd965461fba0d841fd3a2ef81, State: Initialized, Role: FOLLOWER
I20260808 04:45:30.708954 14920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } }
I20260808 04:45:30.709148 14920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:30.709233 14920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:30.709331 14920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:30.710629 14920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } }
I20260808 04:45:30.711017 14920 leader_election.cc:302] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27cbf01bd965461fba0d841fd3a2ef81; no voters:
I20260808 04:45:30.711377 14920 leader_election.cc:288] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:30.711495 14928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:30.711722 14928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 1 LEADER]: Becoming Leader. State: Replica: 27cbf01bd965461fba0d841fd3a2ef81, State: Running, Role: LEADER
I20260808 04:45:30.712050 14928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } }
I20260808 04:45:30.712689 14920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:30.713374 14928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } }
I20260808 04:45:30.713533 14928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:30.713783 14928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27cbf01bd965461fba0d841fd3a2ef81. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } }
I20260808 04:45:30.713896 14928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:30.715997 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:30.700130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.4.217.62:33995)
0808 04:45:30.700605 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:30.700620 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:30.700656 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:45:30.701351 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:30.701369 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:30.701691 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:30.702059 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:30.702898 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:30.703854 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:30.704746 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:30.705274 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:30.705463 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:30.708476 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:30.708482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:30.708499 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:30.708531 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:30.710989 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:30.711544 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:30.711551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:30.711558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:30.711739 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:30.712562 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:30.712565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:30.712567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:30.713136 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:30.713324 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:30.714304 (+ 980us) server_negotiation.cc:299] Negotiation successful
0808 04:45:30.714647 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":151,"threads_started":1}
W20260808 04:45:30.718425 14949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:45:30.718516 14949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:45:30.718606 14936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:30.719334 14936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:30.724911 14936 catalog_manager.cc:1347] Generated new cluster ID: 546263cf6e8a449d960d1a054e92e70e
I20260808 04:45:30.724977 14936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:30.745687 14936 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:30.746627 14936 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:30.752110 14936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: Generated new TSK 0
I20260808 04:45:30.752846 14936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab.
Entry for principal oryx/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab.
I20260808 04:45:30.787871 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33995
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab
--principal=oryx/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:30.922030 14956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:30.922291 14956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:30.922334 14956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:30.926358 14956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:30.926409 14956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:30.926465 14956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:30.931533 14956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab
--principal=oryx/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:30.932473 14956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:30.933498 14956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:45:30.943274 14956 server_base.cc:1034] running on GCE node
W20260808 04:45:30.943244 14966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:30.943284 14963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:30.943442 14964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:30.943912 14956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:30.944514 14956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:30.945693 14956 hybrid_clock.cc:648] HybridClock initialized: now 1786164330945634 us; error 72 us; skew 500 ppm
Aug 08 04:45:30 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164330, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:30.948431 14956 init.cc:377] Logged in from keytab as oryx/127.4.217.1@KRBTEST.COM (short username oryx)
I20260808 04:45:30.949302 14956 webserver.cc:460] Webserver started at http://127.4.217.1:42891/ using document root <none> and password file <none>
I20260808 04:45:30.949841 14956 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:30.949959 14956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:30.950264 14956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:30.952235 14956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "54e274a0492549cb8762a91403cff09d"
format_stamp: "Formatted at 2026-08-08 04:45:30 on dist-test-slave-2kcd"
server_key: "e0301de1240c53960f7e9aa922d97f84"
server_key_iv: "67fa1016d08699d19212b5f4f563d4da"
server_key_version: "encryptionkey@0"
I20260808 04:45:30.952836 14956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "54e274a0492549cb8762a91403cff09d"
format_stamp: "Formatted at 2026-08-08 04:45:30 on dist-test-slave-2kcd"
server_key: "e0301de1240c53960f7e9aa922d97f84"
server_key_iv: "67fa1016d08699d19212b5f4f563d4da"
server_key_version: "encryptionkey@0"
I20260808 04:45:30.956319 14956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:30.958635 14986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:30.959784 14956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:45:30.959924 14956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "54e274a0492549cb8762a91403cff09d"
format_stamp: "Formatted at 2026-08-08 04:45:30 on dist-test-slave-2kcd"
server_key: "e0301de1240c53960f7e9aa922d97f84"
server_key_iv: "67fa1016d08699d19212b5f4f563d4da"
server_key_version: "encryptionkey@0"
I20260808 04:45:30.960032 14956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:30.981822 14956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:30.984515 14956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:30.984707 14956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:30.985308 14956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:30.986101 14956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:30.986171 14956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:30.986239 14956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:30.986280 14956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:30.994830 14956 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:36669
I20260808 04:45:30.994864 15158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:36669 every 8 connection(s)
I20260808 04:45:30.995666 14956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:30.996234 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 14956
I20260808 04:45:30.996383 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:30.996681 4964 external_mini_cluster.cc:1428] Setting key ca1a37cb0e2679bc2554b08308f355ae
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164330, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.4.217.2@KRBTEST.COM; defaulting to no policy
I20260808 04:45:31.010174 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:30.997549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:37001 (local address 127.4.217.62:33995)
0808 04:45:30.997678 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:30.997682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:30.998675 (+ 993us) server_negotiation.cc:407] Connection header received
0808 04:45:30.999951 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:30.999957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.000022 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.000131 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.001766 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.002291 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.003430 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.003685 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.003797 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.006632 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.006635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.006637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.006661 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.008098 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.008612 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.008616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.008618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.008700 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.009075 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.009080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.009082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.009265 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.009372 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.009859 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.009995 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.2@KRBTEST.COM" created.
I20260808 04:45:31.011057 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:30.998000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.1:37001)
0808 04:45:30.998505 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:30.998556 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:30.999749 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:31.000354 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:31.000365 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:31.000920 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:31.001604 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.001627 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.002438 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.002441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.003294 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.003304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.003838 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.003842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.004008 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.004619 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:31.004635 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:31.006463 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:31.008223 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.008229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.008243 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.008497 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.008828 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.008832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.008833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.008970 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.009373 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:31.009380 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:31.009610 (+ 230us) client_negotiation.cc:769] Sending connection context
0808 04:45:31.009785 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 04:45:31.010111 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":137,"threads_started":1}
I20260808 04:45:31.012349 15161 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33995
I20260808 04:45:31.012643 15161 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:31.013279 15161 heartbeater.cc:511] Master 127.4.217.62:33995 requested a full tablet report, sending...
I20260808 04:45:31.014853 14862 ts_manager.cc:194] Registered new tserver with Master: 54e274a0492549cb8762a91403cff09d (127.4.217.1:36669)
I20260808 04:45:31.016026 14862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:37001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab.
Entry for principal oryx/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab.
I20260808 04:45:31.028157 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33995
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab
--principal=oryx/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:31.141621 15178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:31.141899 15178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:31.141973 15178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:31.145939 15178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:31.146030 15178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:31.146119 15178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:31.151342 15178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab
--principal=oryx/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:31.152454 15178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:31.153554 15178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:31.163427 15184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:31.163480 15185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:31.163595 15178 server_base.cc:1034] running on GCE node
W20260808 04:45:31.163517 15188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:31.164126 15178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:31.164865 15178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:31.166060 15178 hybrid_clock.cc:648] HybridClock initialized: now 1786164331166048 us; error 39 us; skew 500 ppm
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:31.168881 15178 init.cc:377] Logged in from keytab as oryx/127.4.217.2@KRBTEST.COM (short username oryx)
I20260808 04:45:31.169773 15178 webserver.cc:460] Webserver started at http://127.4.217.2:44157/ using document root <none> and password file <none>
I20260808 04:45:31.170338 15178 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:31.170400 15178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:31.170635 15178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:31.172721 15178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "51ab903a54744f3eb3b323ad5a297485"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "cb242ac20bcb15e62bba2a2a3aaf9284"
server_key_iv: "5c4a01d71d347caa799974a1d7eb9aec"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.173393 15178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "51ab903a54744f3eb3b323ad5a297485"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "cb242ac20bcb15e62bba2a2a3aaf9284"
server_key_iv: "5c4a01d71d347caa799974a1d7eb9aec"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.177333 15178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:45:31.179726 15202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.180976 15178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:31.181097 15178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "51ab903a54744f3eb3b323ad5a297485"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "cb242ac20bcb15e62bba2a2a3aaf9284"
server_key_iv: "5c4a01d71d347caa799974a1d7eb9aec"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.181193 15178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:31.194751 15178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:31.198189 15178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:31.198405 15178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:31.199069 15178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:31.199949 15178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:31.199997 15178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.200064 15178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:31.200102 15178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.209549 15178 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:43665
I20260808 04:45:31.209573 15375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:43665 every 8 connection(s)
I20260808 04:45:31.210515 15178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:31.214527 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 15178
I20260808 04:45:31.214682 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:31.214986 4964 external_mini_cluster.cc:1428] Setting key e10e00e821e13fcc019000001085b8ae
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:45:31.225610 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.212409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:60301 (local address 127.4.217.62:33995)
0808 04:45:31.212567 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:31.212571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:31.213617 (+ 1046us) server_negotiation.cc:407] Connection header received
0808 04:45:31.215394 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:31.215400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.215461 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.215541 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.216896 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.217719 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.218966 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.219218 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.219339 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.221939 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.221942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.221944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.221969 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.223407 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.223948 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.223952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.223954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.224036 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.224404 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.224407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.224409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.224592 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.224698 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.225310 (+ 612us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.225439 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:45:31.226504 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.212898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.2:60301)
0808 04:45:31.213467 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:31.213512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:31.215113 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:31.215742 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:31.215749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:31.216124 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:31.216708 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.216723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.217846 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.217849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.218808 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.218817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.219380 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.219384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.219563 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.220170 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:31.220188 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:31.221765 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:31.223540 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.223547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.223562 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.223834 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.224128 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.224133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.224138 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.224296 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.224698 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:31.224706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:31.225034 (+ 328us) client_negotiation.cc:769] Sending connection context
0808 04:45:31.225251 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 04:45:31.225603 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":132,"threads_started":1}
I20260808 04:45:31.227711 15377 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33995
I20260808 04:45:31.227967 15377 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.3@KRBTEST.COM" created.
I20260808 04:45:31.228509 15377 heartbeater.cc:511] Master 127.4.217.62:33995 requested a full tablet report, sending...
I20260808 04:45:31.229698 14862 ts_manager.cc:194] Registered new tserver with Master: 51ab903a54744f3eb3b323ad5a297485 (127.4.217.2:43665)
I20260808 04:45:31.230587 14862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:60301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab.
Entry for principal oryx/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab.
I20260808 04:45:31.244380 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33995
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab
--principal=oryx/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:31.358350 15385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:31.358593 15385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:31.358656 15385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:31.362742 15385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:31.362807 15385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:31.362926 15385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:31.368127 15385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab
--principal=oryx/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:31.369107 15385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:31.370038 15385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:31.379830 15399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:31.379866 15396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:31.379922 15385 server_base.cc:1034] running on GCE node
W20260808 04:45:31.379853 15403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:31.380466 15385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:31.381140 15385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:31.382303 15385 hybrid_clock.cc:648] HybridClock initialized: now 1786164331382287 us; error 25 us; skew 500 ppm
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:31.385085 15385 init.cc:377] Logged in from keytab as oryx/127.4.217.3@KRBTEST.COM (short username oryx)
I20260808 04:45:31.385926 15385 webserver.cc:460] Webserver started at http://127.4.217.3:38905/ using document root <none> and password file <none>
I20260808 04:45:31.386417 15385 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:31.386475 15385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:31.386633 15385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:31.388485 15385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "91bc72a32d1c4a729cfee1ebbaeba917"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "69b6a8fd8663c57485c52937e66202a6"
server_key_iv: "8bc60d836c4ff92af475a943fe34b3b9"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.389070 15385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "91bc72a32d1c4a729cfee1ebbaeba917"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "69b6a8fd8663c57485c52937e66202a6"
server_key_iv: "8bc60d836c4ff92af475a943fe34b3b9"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.392591 15385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260808 04:45:31.394843 15418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.395983 15385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:31.396136 15385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "91bc72a32d1c4a729cfee1ebbaeba917"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "69b6a8fd8663c57485c52937e66202a6"
server_key_iv: "8bc60d836c4ff92af475a943fe34b3b9"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.396240 15385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:31.419287 15385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:31.422529 15385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:31.422725 15385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:31.423329 15385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:31.424146 15385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:31.424224 15385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.424294 15385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:31.424345 15385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.433159 15385 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:34263
I20260808 04:45:31.433193 15605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:34263 every 8 connection(s)
I20260808 04:45:31.433995 15385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:31.441589 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 15385
I20260808 04:45:31.441712 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:31.442008 4964 external_mini_cluster.cc:1428] Setting key 439c82d7ac49ef5eafef031dcc48288c
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:31.447356 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.435768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:43661 (local address 127.4.217.62:33995)
0808 04:45:31.435916 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:31.435921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:31.436826 (+ 905us) server_negotiation.cc:407] Connection header received
0808 04:45:31.438108 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:31.438111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.438171 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.438244 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.439399 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.439927 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.441020 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.441252 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.441403 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.444087 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.444090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.444093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.444121 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.445631 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.446095 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.446097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.446099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.446175 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.446469 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.446471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.446472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.446636 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.446712 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.447148 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.447238 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:45:31.448145 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.436178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.3:43661)
0808 04:45:31.436644 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:31.436686 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:31.437877 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:31.438410 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:31.438416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:31.438768 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:31.439261 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.439274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.440031 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.440033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.440912 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.440922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.441450 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.441455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.441620 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.442269 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:31.442286 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:31.443931 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:31.445744 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.445749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.445760 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.446009 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.446253 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.446255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.446256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.446391 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.446725 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:31.446730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:31.446941 (+ 211us) client_negotiation.cc:769] Sending connection context
0808 04:45:31.447101 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 04:45:31.447331 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":127,"threads_started":1}
I20260808 04:45:31.449221 15608 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33995
I20260808 04:45:31.449440 15608 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:31.449870 15608 heartbeater.cc:511] Master 127.4.217.62:33995 requested a full tablet report, sending...
I20260808 04:45:31.450822 14862 ts_manager.cc:194] Registered new tserver with Master: 91bc72a32d1c4a729cfee1ebbaeba917 (127.4.217.3:34263)
I20260808 04:45:31.451351 14862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:43661
I20260808 04:45:31.456358 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.61.keytab.
Entry for principal oryx/127.4.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.61.keytab.
I20260808 04:45:31.485466 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:37733
--webserver_interface=127.4.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.61.keytab
--principal=oryx/127.4.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:33995,127.4.217.61:37733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:31.602442 15615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:31.602716 15615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:31.602762 15615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:31.606765 15615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:31.606824 15615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:31.606843 15615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:31.606860 15615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:31.606884 15615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:31.612246 15615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:33995,127.4.217.61:37733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.61.keytab
--principal=oryx/127.4.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:37733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
W20260808 04:45:31.612437 15615 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.217.62:33995,127.4.217.61:37733'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 04:45:31.613425 15615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:31.614503 15615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:31.622889 15622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:31.622957 15623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:31.622936 15626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:31.623314 15615 server_base.cc:1034] running on GCE node
I20260808 04:45:31.623891 15615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:31.625010 15615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:31.626196 15615 hybrid_clock.cc:648] HybridClock initialized: now 1786164331626180 us; error 25 us; skew 500 ppm
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:31.628950 15615 init.cc:377] Logged in from keytab as oryx/127.4.217.61@KRBTEST.COM (short username oryx)
I20260808 04:45:31.629793 15615 webserver.cc:460] Webserver started at http://127.4.217.61:39881/ using document root <none> and password file <none>
I20260808 04:45:31.630353 15615 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:31.630441 15615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:31.630664 15615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:31.632812 15615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data/instance:
uuid: "a4c3332e4bd74a3bb886772254bc27d1"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "07532a05a188ca17858aa122a17ec834"
server_key_iv: "1ccb906f4cbd430f8b7d6202176b707c"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.633458 15615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal/instance:
uuid: "a4c3332e4bd74a3bb886772254bc27d1"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "07532a05a188ca17858aa122a17ec834"
server_key_iv: "1ccb906f4cbd430f8b7d6202176b707c"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.637223 15615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:31.639549 15642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.640707 15615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:31.640884 15615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal
uuid: "a4c3332e4bd74a3bb886772254bc27d1"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "07532a05a188ca17858aa122a17ec834"
server_key_iv: "1ccb906f4cbd430f8b7d6202176b707c"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.640996 15615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:31.659255 15615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:31.662001 15615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:31.662196 15615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:31.669411 15615 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:37733
I20260808 04:45:31.669685 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.61:37733 every 8 connection(s)
I20260808 04:45:31.670353 15615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
I20260808 04:45:31.672909 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 15615
I20260808 04:45:31.673219 15727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:31.678028 15727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.217.61@KRBTEST.COM
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.61@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:31.687947 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.674386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.4.217.61:37733)
0808 04:45:31.674785 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:31.674799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:31.674836 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:45:31.675815 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:31.675833 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.676180 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.676448 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.677406 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.678457 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.679652 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.680350 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.680523 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.683188 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.683195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.683207 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.683239 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.685056 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.685489 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.685495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.685508 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.685619 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.685961 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.685965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.685966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.686342 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.686554 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.686827 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.687119 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":135,"threads_started":1}
I20260808 04:45:31.689767 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.679981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.4.217.62:33995)
0808 04:45:31.680130 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:31.680136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:31.680457 (+ 321us) server_negotiation.cc:407] Connection header received
0808 04:45:31.680734 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:31.680737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.680848 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.680941 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.681604 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.682174 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.683198 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.683555 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.683671 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.686209 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.686212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.686215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.686247 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.687811 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.688355 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.688360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.688362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.688445 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.688739 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.688743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.688745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.689064 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.689167 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.689560 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.689658 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:45:31.689837 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.679901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:44616)
0808 04:45:31.680326 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:31.680364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:31.680570 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:31.680995 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:31.681001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:31.681342 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:31.681466 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.681471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.682309 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.682312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.683087 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.683097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.683671 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.683676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.683827 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.684473 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:31.684486 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:31.686078 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:31.687941 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.687951 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.687956 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.688257 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.688545 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.688548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.688549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.688645 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.689223 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:31.689229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:31.689344 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 04:45:31.689483 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 04:45:31.689657 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":174,"threads_started":1}
I20260808 04:45:31.691037 15727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.61@KRBTEST.COM for oryx/127.4.217.61@KRBTEST.COM
I20260808 04:45:31.699155 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.691512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.0.0.1:51446)
0808 04:45:31.691610 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:31.691622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:31.691703 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:31.691913 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:31.691916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:31.692066 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:31.692141 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.692146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.692857 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.692860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.693923 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.693932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.694377 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.694380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.694467 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.694892 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:31.694903 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:31.696047 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:31.697777 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.697781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.697783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.697966 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.698232 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.698235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.698237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.698313 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.698647 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:31.698650 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:31.698741 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 04:45:31.698844 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:45:31.698992 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 04:45:31.699187 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.691639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.4.217.61:37733)
0808 04:45:31.691740 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:31.691746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:31.691759 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:31.691789 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:31.691791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.691822 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.691902 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.692247 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.692722 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.694037 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.694273 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.694377 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.696188 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.696192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.696194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.696222 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.697654 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.698060 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.698065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.698067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.698140 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.698395 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.698400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.698401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.698557 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.698656 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.698890 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.699064 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":39}
I20260808 04:45:31.702255 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Bootstrap starting.
I20260808 04:45:31.704463 15727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:31.705255 15727 log.cc:824] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:31.707007 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: No bootstrap required, opened a new log
I20260808 04:45:31.709966 15727 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } }
I20260808 04:45:31.710160 15727 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:31.710237 15727 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c3332e4bd74a3bb886772254bc27d1, State: Initialized, Role: FOLLOWER
I20260808 04:45:31.710667 15727 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } }
I20260808 04:45:31.711110 15739 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } } }
I20260808 04:45:31.711220 15739 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:45:31.711802 15727 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:45:31.717164 15763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:45:31.717252 15763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:45:31.719192 15615 master_runner.cc:428] Detected that this master a4c3332e4bd74a3bb886772254bc27d1 is joining an existing cluster
I20260808 04:45:31.719261 15615 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.217.61:37733
I20260808 04:45:31.729305 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.721293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:44626)
0808 04:45:31.721710 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:31.721725 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:31.721831 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:31.722076 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:31.722080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:31.722306 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:31.722447 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.722455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.723193 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.723197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.724157 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.724169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.724607 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.724610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.724721 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.725229 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:31.725245 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:31.725698 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:31.727697 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.727702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.727704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.727908 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.728195 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.728198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.728200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.728287 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.728666 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:31.728669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:31.728768 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 04:45:31.728918 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 04:45:31.729054 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":106,"threads_started":1}
I20260808 04:45:31.729710 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.721358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.4.217.62:33995)
0808 04:45:31.721491 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:31.721496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:31.721797 (+ 301us) server_negotiation.cc:407] Connection header received
0808 04:45:31.721922 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:31.721924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.721964 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.722029 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.722574 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.723083 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.724284 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.724490 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.724572 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.725843 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.725847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.725851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.725883 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.727586 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.728008 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.728012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.728013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.728101 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.728377 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.728382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.728384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.728567 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.728673 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.729011 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.729138 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:45:31.733616 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.721338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.0.0.1:51448)
0808 04:45:31.725304 (+ 3966us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:31.725316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:31.725433 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:31.725675 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:31.725678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:31.725928 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:31.726097 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.726108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.727073 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.727076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.728200 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.728212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.728923 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.728927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.729030 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.729650 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:31.729661 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:31.730066 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:31.731689 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.731694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.731697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.731944 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.732490 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.732494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.732495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.732601 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.733042 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:31.733046 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:31.733145 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 04:45:31.733272 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 04:45:31.733436 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3908,"thread_start_us":96,"threads_started":1}
I20260808 04:45:31.733628 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.721690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.4.217.61:37733)
0808 04:45:31.721798 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:31.721802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:31.725402 (+ 3600us) server_negotiation.cc:407] Connection header received
0808 04:45:31.725528 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:31.725545 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.725585 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.725661 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.726198 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.726932 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.728330 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.728658 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.728726 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.730190 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.730194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.730196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.730218 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.731569 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.732042 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.732045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.732047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.732383 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.732698 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.732701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.732703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.732937 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.733045 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.733349 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.733500 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":14080}
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164330, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.62@KRBTEST.COM for oryx/127.4.217.61@KRBTEST.COM
I20260808 04:45:31.743798 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.735292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.4.217.61:37733)
0808 04:45:31.735393 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:31.735397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:31.735413 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:45:31.735462 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:31.735464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.735500 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.735562 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.736285 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.736725 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.737951 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.738180 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.738270 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.740682 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.740686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.740692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.740713 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.742087 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.742556 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.742560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.742562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.742639 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.742912 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.742916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.742917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.743093 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.743199 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.743529 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.743647 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 04:45:31.743830 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.734645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.0.0.1:51458)
0808 04:45:31.735017 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:31.735070 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:31.735183 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:31.735619 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:31.735624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:31.736050 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:31.736172 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.736180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.736863 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.736866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.737836 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.737848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.738296 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.738300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.738410 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.738856 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:31.738870 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:31.740555 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:31.742200 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.742212 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.742217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.742454 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.742730 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.742733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.742735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.742820 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.743201 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:31.743210 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:31.743325 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 04:45:31.743474 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:45:31.743666 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20260808 04:45:31.745075 14862 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.217.61:37733: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1"
member_type: NON_VOTER
last_known_addr {
host: "127.4.217.61"
port: 37733
}
attrs {
promote: true
}
}
dest_uuid: "27cbf01bd965461fba0d841fd3a2ef81"
cas_config_opid_index: -1
I20260808 04:45:31.745671 14862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } }
I20260808 04:45:31.747251 15779 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a4c3332e4bd74a3bb886772254bc27d1 (127.4.217.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } } }
I20260808 04:45:31.748062 15687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:31.749205 15779 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.217.61:37733
I20260808 04:45:31.749565 15687 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Refusing update from remote peer 27cbf01bd965461fba0d841fd3a2ef81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 04:45:31.749629 15615 master.cc:537] Master@127.4.217.61:37733 shutting down...
I20260808 04:45:31.751021 15779 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:31.751029 15776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } } }
I20260808 04:45:31.751314 15776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
W20260808 04:45:31.754213 14841 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260808 04:45:31.754397 15615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:31.754658 15615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:31.754779 15615 tablet_replica.cc:330] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: stopping tablet replica
WARNING: no policy specified for oryx/127.4.217.60@KRBTEST.COM; defaulting to no policy
I20260808 04:45:31.761008 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.760694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.0.0.1:51472)
0808 04:45:31.760851 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:31.760935 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:45:31.761206 14841 consensus_peers.cc:487] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 -> Peer a4c3332e4bd74a3bb886772254bc27d1 (127.4.217.61:37733): Couldn't send request to peer a4c3332e4bd74a3bb886772254bc27d1. Status: Network error: Client connection negotiation failed: client connection to 127.4.217.61:37733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.60@KRBTEST.COM" created.
I20260808 04:45:31.770051 15615 master.cc:552] Master@127.4.217.61:37733 shutdown complete.
I20260808 04:45:31.770123 15615 master_runner.cc:305] Clearing existing system tablet
I20260808 04:45:31.770365 15615 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:31.770437 15615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:31.771539 15615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 04:45:31.772399 15792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.772631 15615 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:31.772747 15615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:31.772773 15615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal
uuid: "a4c3332e4bd74a3bb886772254bc27d1"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "07532a05a188ca17858aa122a17ec834"
server_key_iv: "1ccb906f4cbd430f8b7d6202176b707c"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.775403 15615 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:31.778188 15615 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 04:45:31.778265 15615 log.cc:1196] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 04:45:31.778527 15615 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Deleting consensus metadata
I20260808 04:45:31.778698 15615 master_runner.cc:315] Copying system tablet from 127.4.217.62:33995
I20260808 04:45:31.779824 15615 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.217.62:33995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.60.keytab.
Entry for principal oryx/127.4.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.60.keytab.
I20260808 04:45:31.781682 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:44505
--webserver_interface=127.4.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.60.keytab
--principal=oryx/127.4.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:33995,127.4.217.61:37733,127.4.217.60:44505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:45:31.789091 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.780472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.4.217.62:33995)
0808 04:45:31.780617 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:31.780622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:31.780806 (+ 184us) server_negotiation.cc:407] Connection header received
0808 04:45:31.781018 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:31.781022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:31.781079 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:31.781166 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:31.782061 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.782885 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.784118 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.784505 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.784632 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.785948 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:31.785951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:31.785954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:31.785981 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:31.787414 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.787858 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.787862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.787863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.787947 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:31.788231 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:31.788235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:31.788237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:31.788415 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:31.788524 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:31.788820 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 04:45:31.788938 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:45:31.789109 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:31.780379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:44632)
0808 04:45:31.780676 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:31.780692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:31.780850 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:31.781290 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:31.781293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:31.781504 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:31.781891 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.781901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.783048 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.783052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.783973 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:31.783983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:31.784639 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:31.784643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:31.784743 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:31.785385 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:31.785404 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:31.785812 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:31.787539 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.787542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.787544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.787756 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.788049 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:31.788052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:31.788054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:31.788140 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:31.788513 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:31.788516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:31.788631 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 04:45:31.788738 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 04:45:31.788906 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260808 04:45:31.789997 14897 tablet_copy_service.cc:140] P 27cbf01bd965461fba0d841fd3a2ef81: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a4c3332e4bd74a3bb886772254bc27d1 ({username='oryx', principal='oryx/127.4.217.61@KRBTEST.COM'} at 127.0.0.1:44632)
I20260808 04:45:31.790179 14897 tablet_copy_service.cc:161] P 27cbf01bd965461fba0d841fd3a2ef81: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a4c3332e4bd74a3bb886772254bc27d1 at {username='oryx', principal='oryx/127.4.217.61@KRBTEST.COM'} at 127.0.0.1:44632: session id = a4c3332e4bd74a3bb886772254bc27d1-00000000000000000000000000000000
I20260808 04:45:31.791632 14897 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 04:45:31.793236 15615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 04:45:31.795193 15615 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: tablet copy: Starting download of 0 data blocks...
I20260808 04:45:31.795413 15615 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: tablet copy: Starting download of 1 WAL segments...
I20260808 04:45:31.797144 15615 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 04:45:31.798740 14897 tablet_copy_service.cc:342] P 27cbf01bd965461fba0d841fd3a2ef81: Request end of tablet copy session a4c3332e4bd74a3bb886772254bc27d1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.217.61@KRBTEST.COM'} at 127.0.0.1:44632
I20260808 04:45:31.798882 14897 tablet_copy_service.cc:433] P 27cbf01bd965461fba0d841fd3a2ef81: ending tablet copy session a4c3332e4bd74a3bb886772254bc27d1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a4c3332e4bd74a3bb886772254bc27d1
I20260808 04:45:31.805676 15615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:31.807430 15809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:31.807614 15805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:31.807703 15615 server_base.cc:1034] running on GCE node
W20260808 04:45:31.807822 15806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:31.808045 15615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:31.808358 15615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:31.809500 15615 hybrid_clock.cc:648] HybridClock initialized: now 1786164331809512 us; error 47 us; skew 500 ppm
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:31.810933 15615 init.cc:377] Logged in from keytab as oryx/127.4.217.61@KRBTEST.COM (short username oryx)
I20260808 04:45:31.811409 15615 webserver.cc:460] Webserver started at http://127.4.217.61:33565/ using document root <none> and password file <none>
I20260808 04:45:31.811607 15615 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:31.811661 15615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:31.812769 15615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:31.813787 15822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.814076 15615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 04:45:31.814162 15615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal
uuid: "a4c3332e4bd74a3bb886772254bc27d1"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "07532a05a188ca17858aa122a17ec834"
server_key_iv: "1ccb906f4cbd430f8b7d6202176b707c"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.814253 15615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:31.828110 15615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:31.830159 15615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:31.835176 15615 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:37733
I20260808 04:45:31.835450 15909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.61:37733 every 8 connection(s)
I20260808 04:45:31.835865 15615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
I20260808 04:45:31.837587 15910 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:45:31.838423 15910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Bootstrap starting.
I20260808 04:45:31.846956 15910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:45:31.847267 15910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Bootstrap complete.
I20260808 04:45:31.847801 15910 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } }
I20260808 04:45:31.847941 15910 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4c3332e4bd74a3bb886772254bc27d1, State: Initialized, Role: LEARNER
I20260808 04:45:31.848078 15910 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } }
I20260808 04:45:31.848415 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } } }
I20260808 04:45:31.848596 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:45:31.848603 15910 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:31.851711 15939 catalog_manager.cc:1259] Loaded cluster ID: 546263cf6e8a449d960d1a054e92e70e
I20260808 04:45:31.851810 15939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: loading cluster ID for follower catalog manager: success
I20260808 04:45:31.853570 15939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: acquiring CA information for follower catalog manager: success
I20260808 04:45:31.854684 15939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 04:45:31.938818 15803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:31.939070 15803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:31.939110 15803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:31.942960 15803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:31.943011 15803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:31.943032 15803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:31.943048 15803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:31.943063 15803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:31.948052 15803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:33995,127.4.217.61:37733,127.4.217.60:44505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.60.keytab
--principal=oryx/127.4.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:44505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:31.949123 15803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:31.950119 15803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:31.958400 15950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:31.958392 15952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:31.958391 15954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:31.958686 15803 server_base.cc:1034] running on GCE node
I20260808 04:45:31.959257 15803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:31.960275 15803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:31.961457 15803 hybrid_clock.cc:648] HybridClock initialized: now 1786164331961425 us; error 50 us; skew 500 ppm
Aug 08 04:45:31 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:31.964352 15803 init.cc:377] Logged in from keytab as oryx/127.4.217.60@KRBTEST.COM (short username oryx)
I20260808 04:45:31.965397 15803 webserver.cc:460] Webserver started at http://127.4.217.60:37599/ using document root <none> and password file <none>
I20260808 04:45:31.965960 15803 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:31.966063 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:31.966276 15803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:31.968400 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data/instance:
uuid: "5d047bb613c04b4697dc7bbacc315424"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "0d57d9e917ce8250ee88794fcec94197"
server_key_iv: "f1b167700bf40e7ee9e326190cf3167a"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.969046 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal/instance:
uuid: "5d047bb613c04b4697dc7bbacc315424"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "0d57d9e917ce8250ee88794fcec94197"
server_key_iv: "f1b167700bf40e7ee9e326190cf3167a"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.972620 15803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:45:31.974895 15969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:31.976028 15803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:31.976183 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal
uuid: "5d047bb613c04b4697dc7bbacc315424"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "0d57d9e917ce8250ee88794fcec94197"
server_key_iv: "f1b167700bf40e7ee9e326190cf3167a"
server_key_version: "encryptionkey@0"
I20260808 04:45:31.976289 15803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:31.999884 15803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:32.003041 15803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:32.003238 15803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:32.012420 15803 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.60:44505
I20260808 04:45:32.012699 16051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.60:44505 every 8 connection(s)
I20260808 04:45:32.013526 15803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
I20260808 04:45:32.016616 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:32.018728 15161 heartbeater.cc:503] Master 127.4.217.62:33995 was elected leader, sending a full tablet report...
I20260808 04:45:32.019793 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 15803
I20260808 04:45:32.021054 16052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.60@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.217.60@KRBTEST.COM
I20260808 04:45:32.034184 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.022868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.4.217.62:33995)
0808 04:45:32.023006 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.023012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.023322 (+ 310us) server_negotiation.cc:407] Connection header received
0808 04:45:32.023581 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.023585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.023635 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.023724 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.024601 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.025407 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.026845 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.027181 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.027293 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.030316 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.030323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.030329 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.030366 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.031936 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.032478 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.032480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.032482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.032562 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.032954 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.032957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.032958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.033193 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.033281 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.033875 (+ 594us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.034009 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:45:32.035130 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.022736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:44648)
0808 04:45:32.023184 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.023225 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.023419 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.023817 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.023825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.024288 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.024467 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.024477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.025526 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.025528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.026719 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.026734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.027318 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.027321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.027484 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.028238 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.028260 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.030096 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.032046 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.032053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.032062 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.032377 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.032653 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.032656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.032658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.032856 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.033297 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.033302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.033614 (+ 312us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.033823 (+ 209us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.034120 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":190,"threads_started":1}
I20260808 04:45:32.035130 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.021361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.4.217.60:44505)
0808 04:45:32.021777 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.021800 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.021835 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:45:32.022580 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.022606 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.023042 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.023359 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.024271 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.025632 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.026778 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.027341 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.027478 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.030724 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.030732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.030745 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.030776 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.032679 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.033202 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.033207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.033208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.033303 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.033577 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.033580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.033582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.033850 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.034022 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.034273 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.034399 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":136,"threads_started":1}
I20260808 04:45:32.036357 16052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.60@KRBTEST.COM for oryx/127.4.217.61@KRBTEST.COM
I20260808 04:45:32.044014 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.036850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.0.0.1:51484)
0808 04:45:32.037021 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.037036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.037129 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.037481 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.037483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.037607 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.037691 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.037695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.038523 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.038526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.039032 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.039039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.039453 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.039455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.039565 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.039997 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.040015 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.041170 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.042834 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.042837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.042838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.043019 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.043271 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.043274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.043275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.043346 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.043669 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.043671 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.043741 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.043818 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.043908 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260808 04:45:32.044093 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.036973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.4.217.61:37733)
0808 04:45:32.037258 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.037263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.037278 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:32.037322 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.037324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.037374 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.037457 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.037793 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.038426 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.039138 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.039361 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.039438 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.041307 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.041310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.041312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.041335 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.042735 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.043115 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.043117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.043118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.043191 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.043421 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.043423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.043424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.043591 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.043667 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.043872 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.043968 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":101,"threads_started":1}
I20260808 04:45:32.046010 16052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164331, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.60@KRBTEST.COM for oryx/127.4.217.60@KRBTEST.COM
I20260808 04:45:32.054128 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.046426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.0.0.1:42622)
0808 04:45:32.046534 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.046548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.046623 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.046827 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.046830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.046992 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.047051 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.047056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.047737 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.047743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.048229 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.048236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.049200 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.049203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.049275 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.049735 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.049747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.051070 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.052742 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.052745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.052746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.053035 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.053278 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.053280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.053281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.053355 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.053701 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.053704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.053786 (+ 82us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.053891 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.053989 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 04:45:32.054162 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.046517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.4.217.60:44505)
0808 04:45:32.046608 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.046612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.046624 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:45:32.046705 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.046713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.046743 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.046816 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.047153 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.047635 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.048336 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.049091 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.049201 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.051197 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.051200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.051202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.051224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.052645 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.053122 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.053125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.053126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.053198 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.053431 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.053433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.053434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.053592 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.053668 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.053941 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.054042 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":37}
I20260808 04:45:32.057236 16052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: Bootstrap starting.
I20260808 04:45:32.059546 16052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:32.060230 16052 log.cc:824] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:32.062108 16052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: No bootstrap required, opened a new log
I20260808 04:45:32.065438 16052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } }
I20260808 04:45:32.065685 16052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:32.065762 16052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d047bb613c04b4697dc7bbacc315424, State: Initialized, Role: FOLLOWER
I20260808 04:45:32.066195 16052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } }
I20260808 04:45:32.066692 16064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } } }
I20260808 04:45:32.066833 16064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:45:32.067425 16052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:45:32.072290 16084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:45:32.072381 16084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:45:32.075750 15803 master_runner.cc:428] Detected that this master 5d047bb613c04b4697dc7bbacc315424 is joining an existing cluster
I20260808 04:45:32.075814 15803 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.217.60:44505
I20260808 04:45:32.088557 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 14956
I20260808 04:45:32.088855 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.078780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.0.0.1:42632)
0808 04:45:32.079497 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.079528 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.079660 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.080021 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.080023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.080215 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.080312 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.080317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.083065 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.083068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.083865 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.083878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.084385 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.084387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.084445 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.084952 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.084965 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.085306 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.087160 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.087164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.087166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.087421 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.087716 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.087721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.087723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.087820 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.088275 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.088278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.088362 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.088478 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.088640 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":126,"threads_started":1}
I20260808 04:45:32.091526 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.078830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.4.217.60:44505)
0808 04:45:32.078946 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.078950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.079616 (+ 666us) server_negotiation.cc:407] Connection header received
0808 04:45:32.079790 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.079794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.079832 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.079886 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.080438 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.081294 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.083977 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.084222 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.084277 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.085463 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.085467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.085471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.085508 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.087055 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.087532 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.087535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.087537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.087631 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.087909 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.087913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.087914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.088149 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.088242 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.088547 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.088753 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":50}
I20260808 04:45:32.101085 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:36669
--local_ip_for_outbound_sockets=127.4.217.1
--tserver_master_addrs=127.4.217.62:33995
--webserver_port=42891
--webserver_interface=127.4.217.1
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab
--principal=oryx/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.217.62:33995,127.4.217.61:37733
--tserver_master_addrs=127.4.217.62:33995,127.4.217.61:37733,127.4.217.60:44505 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:45:32.113269 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.078382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.4.217.62:33995)
0808 04:45:32.078534 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.078539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.079111 (+ 572us) server_negotiation.cc:407] Connection header received
0808 04:45:32.079323 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.079326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.079399 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.079479 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.080626 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.081338 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.083058 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.083413 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.083531 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.109449 (+ 25918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.109454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.109457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.109491 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.111240 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.111769 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.111773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.111774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.111859 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.112158 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.112162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.112164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.112419 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.112495 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.112934 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.113069 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:45:32.113940 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.078285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:44662)
0808 04:45:32.079027 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.079041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.079208 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.079631 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.079634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.080026 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.080485 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.080490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.081498 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.081502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.082914 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.082924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.083667 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.083671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.083763 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.084364 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.084379 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.109228 (+ 24849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.111378 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.111381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.111384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.111650 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.111969 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.111972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.111974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.112058 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.112600 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.112603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.112709 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.113514 (+ 805us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.113716 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":72,"spinlock_wait_cycles":20480,"thread_start_us":109,"threads_started":1}
I20260808 04:45:32.115906 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.078285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.0.0.1:51500)
0808 04:45:32.078717 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.078734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.078828 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.079708 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.079713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.080028 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.080295 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.080304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.082358 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.082361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.083228 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.083238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.083918 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.083924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.084039 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.109406 (+ 25367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.109430 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.111096 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.113875 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.113879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.113881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.114147 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.114795 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.114798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.114800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.114892 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.115364 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.115367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.115481 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.115579 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.115719 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":23,"spinlock_wait_cycles":16256,"thread_start_us":125,"threads_started":1}
I20260808 04:45:32.117190 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.079185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.4.217.61:37733)
0808 04:45:32.079313 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.079317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.079331 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:45:32.079372 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.079376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.079420 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.079557 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.081441 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.082224 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.083360 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.083708 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.083797 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.111331 (+ 27534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.111337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.111340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.111382 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.113451 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.114512 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.114517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.114522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.114608 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.115007 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.115010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.115012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.115228 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.115319 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.116674 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.116860 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164330, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.62@KRBTEST.COM for oryx/127.4.217.60@KRBTEST.COM
I20260808 04:45:32.127962 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.118937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.0.0.1:42634)
0808 04:45:32.119068 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.119084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.119169 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.119437 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.119440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.119627 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.119731 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.119738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.120809 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.120813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.121679 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.121690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.122288 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.122292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.122391 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.122869 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.122883 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.124434 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.126392 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.126395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.126397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.126645 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.126943 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.126946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.126948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.127051 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.127456 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.127459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.127555 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.127664 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.127800 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 04:45:32.127982 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.119035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.4.217.60:44505)
0808 04:45:32.119165 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.119169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.119184 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:32.119256 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.119259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.119309 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.119395 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.119846 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.120653 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.121797 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.122182 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.122282 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.124593 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.124597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.124600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.124625 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.126266 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.126752 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.126756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.126758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.126841 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.127152 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.127155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.127157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.127358 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.127455 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.127728 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.127840 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:45:32.129441 14862 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.217.60:44505: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5d047bb613c04b4697dc7bbacc315424"
member_type: NON_VOTER
last_known_addr {
host: "127.4.217.60"
port: 44505
}
attrs {
promote: true
}
}
dest_uuid: "27cbf01bd965461fba0d841fd3a2ef81"
cas_config_opid_index: 5
I20260808 04:45:32.130052 14862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } }
I20260808 04:45:32.131105 15776 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5d047bb613c04b4697dc7bbacc315424 (127.4.217.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.132145 16007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:32.132638 15776 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.217.60:44505
I20260808 04:45:32.132828 15776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.132921 15776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:32.133556 15803 master.cc:537] Master@127.4.217.60:44505 shutting down...
I20260808 04:45:32.133628 16007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 FOLLOWER]: Refusing update from remote peer 27cbf01bd965461fba0d841fd3a2ef81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 04:45:32.138011 15784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:45:32.140236 14841 consensus_peers.cc:487] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 -> Peer 5d047bb613c04b4697dc7bbacc315424 (127.4.217.60:44505): Couldn't send request to peer 5d047bb613c04b4697dc7bbacc315424. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 04:45:32.144411 15803 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:32.144712 15803 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:32.144872 15803 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: stopping tablet replica
I20260808 04:45:32.146023 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.131328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.0.0.1:51504)
0808 04:45:32.141103 (+ 9775us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.141117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.141223 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.141642 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.141645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.141662 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:32.141742 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.141748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.142788 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.142792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.144019 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.144037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.145600 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.145604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.145699 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.145743 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.145817 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.145889 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9708}
I20260808 04:45:32.146441 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.131546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.4.217.61:37733)
0808 04:45:32.141011 (+ 9465us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.141016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.141307 (+ 291us) server_negotiation.cc:407] Connection header received
0808 04:45:32.141362 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.141365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.141425 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.141518 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:32.141861 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.142656 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.144356 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.145246 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.145346 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.145520 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.146248 (+ 728us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.146324 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8258}
I20260808 04:45:32.147208 15864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 LEARNER]: Refusing update from remote peer 27cbf01bd965461fba0d841fd3a2ef81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 04:45:32.147930 15784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 04:45:32.149979 15864 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5d047bb613c04b4697dc7bbacc315424 (127.4.217.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.152094 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27cbf01bd965461fba0d841fd3a2ef81. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.152216 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:45:32.152295 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: NON_VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: true } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.152369 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:45:32.153532 15784 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: attempting to promote NON_VOTER a4c3332e4bd74a3bb886772254bc27d1 to VOTER
I20260808 04:45:32.154014 15784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } }
I20260808 04:45:32.155831 15803 master.cc:552] Master@127.4.217.60:44505 shutdown complete.
I20260808 04:45:32.155928 15803 master_runner.cc:305] Clearing existing system tablet
I20260808 04:45:32.156224 15803 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:32.156318 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:32.157663 15803 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:32.157943 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.157537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.0.0.1:42640)
0808 04:45:32.157803 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.157863 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:44505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260808 04:45:32.158619 16101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.158704 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.158434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.0.0.1:42652)
0808 04:45:32.158585 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.158637 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:44505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260808 04:45:32.158831 14841 consensus_peers.cc:487] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 -> Peer 5d047bb613c04b4697dc7bbacc315424 (127.4.217.60:44505): Couldn't send request to peer 5d047bb613c04b4697dc7bbacc315424. Status: Network error: Client connection negotiation failed: client connection to 127.4.217.60:44505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:45:32.158979 15803 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:32.159099 15803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:32.159133 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal
uuid: "5d047bb613c04b4697dc7bbacc315424"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "0d57d9e917ce8250ee88794fcec94197"
server_key_iv: "f1b167700bf40e7ee9e326190cf3167a"
server_key_version: "encryptionkey@0"
I20260808 04:45:32.159344 15864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 LEARNER]: Refusing update from remote peer 27cbf01bd965461fba0d841fd3a2ef81: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 04:45:32.159727 15779 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 04:45:32.161274 15803 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:32.163713 15803 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 04:45:32.163797 15803 log.cc:1196] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 04:45:32.164041 15803 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: Deleting consensus metadata
I20260808 04:45:32.164356 15803 master_runner.cc:315] Copying system tablet from 127.4.217.62:33995
I20260808 04:45:32.165659 15803 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: tablet copy: Beginning tablet copy session from remote peer at address 127.4.217.62:33995
I20260808 04:45:32.166841 15779 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a4c3332e4bd74a3bb886772254bc27d1 (127.4.217.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.168421 15779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.168570 15779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:32.168702 15779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.169720 15862 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a4c3332e4bd74a3bb886772254bc27d1 (127.4.217.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.171765 16098 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.171893 16098 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:45:32.172289 15779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:32.177412 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.166769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.4.217.62:33995)
0808 04:45:32.166902 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.166906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.167602 (+ 696us) server_negotiation.cc:407] Connection header received
0808 04:45:32.167795 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.167798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.167856 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.167918 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.168947 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.169777 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.171121 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.171459 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.171534 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.173290 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.173294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.173298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.173325 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.175041 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.175598 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.175602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.175604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.175698 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.176087 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.176091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.176093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.176348 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.176470 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.177121 (+ 651us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.177251 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:45:32.177452 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.167157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:44668)
0808 04:45:32.167513 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.167529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.167686 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.168072 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.168075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.168304 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.168660 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.168677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.170064 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.170067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.170974 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.170986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.171706 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.171709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.171811 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.172606 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.172627 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.173125 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.175166 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.175170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.175172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.175469 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.175838 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.175842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.175844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.175969 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.176743 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.176746 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.176889 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.177046 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.177207 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":77,"threads_started":1}
I20260808 04:45:32.178505 14897 tablet_copy_service.cc:140] P 27cbf01bd965461fba0d841fd3a2ef81: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5d047bb613c04b4697dc7bbacc315424 ({username='oryx', principal='oryx/127.4.217.60@KRBTEST.COM'} at 127.0.0.1:44668)
I20260808 04:45:32.178610 14897 tablet_copy_service.cc:161] P 27cbf01bd965461fba0d841fd3a2ef81: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5d047bb613c04b4697dc7bbacc315424 at {username='oryx', principal='oryx/127.4.217.60@KRBTEST.COM'} at 127.0.0.1:44668: session id = 5d047bb613c04b4697dc7bbacc315424-00000000000000000000000000000000
I20260808 04:45:32.179699 14897 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 04:45:32.181244 15803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 04:45:32.183655 15803 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: tablet copy: Starting download of 0 data blocks...
I20260808 04:45:32.183856 15803 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: tablet copy: Starting download of 1 WAL segments...
I20260808 04:45:32.185498 15803 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 04:45:32.187443 14897 tablet_copy_service.cc:342] P 27cbf01bd965461fba0d841fd3a2ef81: Request end of tablet copy session 5d047bb613c04b4697dc7bbacc315424-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.217.60@KRBTEST.COM'} at 127.0.0.1:44668
I20260808 04:45:32.187536 14897 tablet_copy_service.cc:433] P 27cbf01bd965461fba0d841fd3a2ef81: ending tablet copy session 5d047bb613c04b4697dc7bbacc315424-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5d047bb613c04b4697dc7bbacc315424
I20260808 04:45:32.195782 15803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:32.197678 16108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:32.201766 16113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:32.202052 16117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:32.202267 15803 server_base.cc:1034] running on GCE node
I20260808 04:45:32.202567 15803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:32.202998 15803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:32.204195 15803 hybrid_clock.cc:648] HybridClock initialized: now 1786164332204179 us; error 32 us; skew 500 ppm
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:32.205911 15803 init.cc:377] Logged in from keytab as oryx/127.4.217.60@KRBTEST.COM (short username oryx)
I20260808 04:45:32.206403 15803 webserver.cc:460] Webserver started at http://127.4.217.60:36059/ using document root <none> and password file <none>
I20260808 04:45:32.206594 15803 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:32.206647 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:32.207990 15803 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:32.209177 16125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.212332 15803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260808 04:45:32.212430 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal
uuid: "5d047bb613c04b4697dc7bbacc315424"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "0d57d9e917ce8250ee88794fcec94197"
server_key_iv: "f1b167700bf40e7ee9e326190cf3167a"
server_key_version: "encryptionkey@0"
I20260808 04:45:32.212515 15803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:32.227228 15803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:32.229665 15803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:32.232762 15377 heartbeater.cc:503] Master 127.4.217.62:33995 was elected leader, sending a full tablet report...
I20260808 04:45:32.234707 15803 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.60:44505
I20260808 04:45:32.235356 15803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
I20260808 04:45:32.235638 16206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.60:44505 every 8 connection(s)
I20260808 04:45:32.237835 16207 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:45:32.238897 16207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: Bootstrap starting.
I20260808 04:45:32.254967 16207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:45:32.255573 16207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: Bootstrap complete.
I20260808 04:45:32.256573 16207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } }
I20260808 04:45:32.256947 16207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d047bb613c04b4697dc7bbacc315424, State: Initialized, Role: LEARNER
W20260808 04:45:32.256949 16095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:32.257299 16095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:32.257391 16095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 04:45:32.257211 16207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } }
I20260808 04:45:32.258222 16207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:32.259438 16219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.259585 16219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: This master's current role is: LEARNER
W20260808 04:45:32.263420 16095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:32.263495 16095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:32.263607 16095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:32.263988 16237 catalog_manager.cc:1259] Loaded cluster ID: 546263cf6e8a449d960d1a054e92e70e
I20260808 04:45:32.264042 16237 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: loading cluster ID for follower catalog manager: success
I20260808 04:45:32.266129 16237 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: acquiring CA information for follower catalog manager: success
I20260808 04:45:32.267335 16237 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:45:32.269565 16095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab
--principal=oryx/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:36669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=42891
--tserver_master_addrs=127.4.217.62:33995,127.4.217.61:37733,127.4.217.60:44505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:32.270565 16095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:32.271584 16095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:32.281378 16239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:32.281340 16243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:32.281426 16095 server_base.cc:1034] running on GCE node
W20260808 04:45:32.281376 16240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:32.282102 16095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:32.282706 16095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:32.283865 16095 hybrid_clock.cc:648] HybridClock initialized: now 1786164332283832 us; error 42 us; skew 500 ppm
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:32.286643 16095 init.cc:377] Logged in from keytab as oryx/127.4.217.1@KRBTEST.COM (short username oryx)
I20260808 04:45:32.287420 16095 webserver.cc:460] Webserver started at http://127.4.217.1:42891/ using document root <none> and password file <none>
I20260808 04:45:32.287961 16095 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:32.288066 16095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:32.291790 16095 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:32.293824 16253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.294976 16095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:32.295111 16095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "54e274a0492549cb8762a91403cff09d"
format_stamp: "Formatted at 2026-08-08 04:45:30 on dist-test-slave-2kcd"
server_key: "e0301de1240c53960f7e9aa922d97f84"
server_key_iv: "67fa1016d08699d19212b5f4f563d4da"
server_key_version: "encryptionkey@0"
I20260808 04:45:32.295631 16095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:32.305763 16095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:32.308375 16095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:32.308575 16095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:32.309185 16095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:32.309996 16095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:32.310066 16095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.310135 16095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:32.310175 16095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.321074 16095 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:36669
I20260808 04:45:32.321097 16433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:36669 every 8 connection(s)
I20260808 04:45:32.322009 16095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:32.323895 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 16095
I20260808 04:45:32.324142 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 15178
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.60@KRBTEST.COM
I20260808 04:45:32.342563 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.323831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:38343 (local address 127.4.217.62:33995)
0808 04:45:32.323962 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.323966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.328981 (+ 5015us) server_negotiation.cc:407] Connection header received
0808 04:45:32.329127 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.329130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.329181 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.329232 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.330925 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.331761 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.333858 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.334192 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.334267 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.337877 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.337881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.337885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.337915 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.339913 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.340568 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.340574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.340579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.340673 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.341128 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.341135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.341139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.341400 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.341486 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.342215 (+ 729us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.342357 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:45:32.344580 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.324290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.1:38343)
0808 04:45:32.328894 (+ 4604us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.328908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.329009 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.329374 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.329380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.329793 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.330564 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.330584 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.332159 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.332162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.333724 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.333740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.334459 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.334463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.334687 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.335351 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.335374 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.337665 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.340074 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.340082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.340100 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.340434 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.340814 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.340820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.340824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.341013 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.341570 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.341579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.341917 (+ 338us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.343061 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.343431 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4314,"mutex_wait_us":499,"spinlock_wait_cycles":1920,"thread_start_us":3697,"threads_started":1}
I20260808 04:45:32.345890 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.324175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:41605 (local address 127.4.217.60:44505)
0808 04:45:32.324495 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.324500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.327314 (+ 2814us) server_negotiation.cc:407] Connection header received
0808 04:45:32.328558 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.328574 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.328640 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.328711 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.330882 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.331963 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.334731 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.335127 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.335221 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.338654 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.338658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.338661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.338691 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.343717 (+ 5026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.344218 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.344222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.344223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.344317 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.344629 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.344634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.344639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.344893 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.344987 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.345588 (+ 601us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.345725 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":88,"threads_started":1}
I20260808 04:45:32.345968 16436 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33995
I20260808 04:45:32.346215 16436 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:32.346760 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.324290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.4.217.1:41605)
0808 04:45:32.327174 (+ 2884us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.327225 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.328319 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.329230 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.329238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.329940 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.330777 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.330783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.332158 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.332162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.334609 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.334617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.335404 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.335410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.335529 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.336861 (+ 1332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.336881 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.338479 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.343848 (+ 5369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.343851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.343853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.344096 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.344428 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.344430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.344432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.344517 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.345287 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.345289 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.345385 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.346445 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.346592 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2669,"mutex_wait_us":1676,"spinlock_wait_cycles":3968,"thread_start_us":182,"threads_started":1}
I20260808 04:45:32.346810 16436 heartbeater.cc:511] Master 127.4.217.62:33995 requested a full tablet report, sending...
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.61@KRBTEST.COM
I20260808 04:45:32.349563 14862 ts_manager.cc:194] Re-registered known tserver with Master: 54e274a0492549cb8762a91403cff09d (127.4.217.1:36669)
I20260808 04:45:32.350800 16437 heartbeater.cc:347] Connected to a master server at 127.4.217.60:44505
I20260808 04:45:32.350891 16437 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:32.351017 14862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:38343
I20260808 04:45:32.351115 16437 heartbeater.cc:511] Master 127.4.217.60:44505 requested a full tablet report, sending...
I20260808 04:45:32.353248 16150 ts_manager.cc:194] Registered new tserver with Master: 54e274a0492549cb8762a91403cff09d (127.4.217.1:36669)
I20260808 04:45:32.356053 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.328772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:38859 (local address 127.4.217.61:37733)
0808 04:45:32.329079 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.329087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.329832 (+ 745us) server_negotiation.cc:407] Connection header received
0808 04:45:32.330443 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.330446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.330496 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.330550 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.333775 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.334513 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.339069 (+ 4556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.339492 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.339578 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.349814 (+ 10236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.349819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.349822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.349855 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.352083 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.354248 (+ 2165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.354255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.354260 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.354361 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.354675 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.354681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.354686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.354939 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.355064 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.355595 (+ 531us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.355782 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260808 04:45:32.357116 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.325580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.4.217.1:38859)
0808 04:45:32.329745 (+ 4165us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.329761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.330306 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.330701 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.330703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.332580 (+ 1877us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.332691 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.332699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.335768 (+ 3069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.335773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.336644 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.336657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.347130 (+ 10473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.347133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.347209 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.347883 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.347895 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.349571 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.353853 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.353856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.353858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.354126 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.354465 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.354471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.354475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.354575 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.355146 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.355151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.355309 (+ 158us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.356271 (+ 962us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.356476 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4100,"thread_start_us":2836,"threads_started":1}
I20260808 04:45:32.359549 16438 heartbeater.cc:347] Connected to a master server at 127.4.217.61:37733
I20260808 04:45:32.359661 16438 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:32.359938 16438 heartbeater.cc:511] Master 127.4.217.61:37733 requested a full tablet report, sending...
I20260808 04:45:32.361145 15849 ts_manager.cc:194] Registered new tserver with Master: 54e274a0492549cb8762a91403cff09d (127.4.217.1:36669)
I20260808 04:45:32.363682 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:43665
--local_ip_for_outbound_sockets=127.4.217.2
--tserver_master_addrs=127.4.217.62:33995
--webserver_port=44157
--webserver_interface=127.4.217.2
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab
--principal=oryx/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.217.62:33995,127.4.217.61:37733
--tserver_master_addrs=127.4.217.62:33995,127.4.217.61:37733,127.4.217.60:44505 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:45:32.453274 15608 heartbeater.cc:503] Master 127.4.217.62:33995 was elected leader, sending a full tablet report...
W20260808 04:45:32.478509 16454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:32.478739 16454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:32.478778 16454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:32.482686 16454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:32.482745 16454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:32.482801 16454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:32.487740 16454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab
--principal=oryx/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:43665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=44157
--tserver_master_addrs=127.4.217.62:33995,127.4.217.61:37733,127.4.217.60:44505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:32.488693 16454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:32.489722 16454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:32.499316 16468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:32.499327 16463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:32.499634 16454 server_base.cc:1034] running on GCE node
W20260808 04:45:32.499365 16464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:32.500185 16454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:32.501256 16454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:32.502501 16454 hybrid_clock.cc:648] HybridClock initialized: now 1786164332502486 us; error 29 us; skew 500 ppm
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:32.505579 16454 init.cc:377] Logged in from keytab as oryx/127.4.217.2@KRBTEST.COM (short username oryx)
I20260808 04:45:32.506459 16454 webserver.cc:460] Webserver started at http://127.4.217.2:44157/ using document root <none> and password file <none>
I20260808 04:45:32.507021 16454 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:32.507138 16454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:32.511011 16454 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:45:32.513240 16481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.514722 16454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:45:32.514856 16454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "51ab903a54744f3eb3b323ad5a297485"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "cb242ac20bcb15e62bba2a2a3aaf9284"
server_key_iv: "5c4a01d71d347caa799974a1d7eb9aec"
server_key_version: "encryptionkey@0"
I20260808 04:45:32.515380 16454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:32.524616 16454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:32.527252 16454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:32.527457 16454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:32.528044 16454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:32.528964 16454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:32.529016 16454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.529081 16454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:32.529114 16454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.538519 16454 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:43665
I20260808 04:45:32.538553 16674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:43665 every 8 connection(s)
I20260808 04:45:32.539548 16454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:32.540493 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 16454
I20260808 04:45:32.540689 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 15385
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.60@KRBTEST.COM
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.61@KRBTEST.COM
I20260808 04:45:32.555169 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:34263
--local_ip_for_outbound_sockets=127.4.217.3
--tserver_master_addrs=127.4.217.62:33995
--webserver_port=38905
--webserver_interface=127.4.217.3
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab
--principal=oryx/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.217.62:33995,127.4.217.61:37733
--tserver_master_addrs=127.4.217.62:33995,127.4.217.61:37733,127.4.217.60:44505 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:32.560730 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.542542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:50781 (local address 127.4.217.61:37733)
0808 04:45:32.542670 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.542675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.546255 (+ 3580us) server_negotiation.cc:407] Connection header received
0808 04:45:32.547789 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.547796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.547864 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.547940 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.549906 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.550798 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.552250 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.552648 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.552766 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.555825 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.555832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.555838 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.555874 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.557818 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.558501 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.558504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.558507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.558602 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.559120 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.559125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.559127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.559420 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.559524 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.560385 (+ 861us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.560520 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:45:32.561975 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.545933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.4.217.2:48703)
0808 04:45:32.546843 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.546859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.547433 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.548161 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.548171 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.548914 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.549904 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.549910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.550671 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.550675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.551619 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.551629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.552152 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.552157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.552338 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.553007 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.553047 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.555590 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.557819 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.557827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.557845 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.558397 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.558776 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.558782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.558783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.558992 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.559630 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.559639 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.560062 (+ 423us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.560298 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.560697 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":106,"spinlock_wait_cycles":31360,"thread_start_us":81,"threads_started":1}
I20260808 04:45:32.562264 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.542156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.4.217.2:50781)
0808 04:45:32.546089 (+ 3933us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.546150 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.547441 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.548161 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.548173 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.548923 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.549735 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.549752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.551013 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.551019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.552096 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.552112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.552821 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.552827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.552961 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.553455 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.553470 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.555638 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.557952 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.557957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.557962 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.558391 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.558753 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.558762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.558765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.558978 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.559636 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.559639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.560065 (+ 426us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.560969 (+ 904us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.561137 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3679,"mutex_wait_us":102,"spinlock_wait_cycles":36480,"thread_start_us":1092,"threads_started":1}
I20260808 04:45:32.562289 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.541418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:43111 (local address 127.4.217.62:33995)
0808 04:45:32.542323 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.542328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.546329 (+ 4001us) server_negotiation.cc:407] Connection header received
0808 04:45:32.547700 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.547703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.547761 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.547816 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.552961 (+ 5145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.553831 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.555086 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.555450 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.555538 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.558491 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.558494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.558497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.558526 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.560284 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.560851 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.560855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.560856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.560965 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.561298 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.561301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.561303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.561514 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.561612 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.561945 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.562093 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794}
I20260808 04:45:32.562744 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.542156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.2:43111)
0808 04:45:32.546214 (+ 4058us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.546232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.547433 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.548161 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.548171 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.548930 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.549740 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.549754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.554025 (+ 4271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.554030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.554917 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.554929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.555725 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.555732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.555824 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.556697 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.556712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.558316 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.560412 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.560415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.560416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.560702 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.561084 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.561088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.561093 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.561181 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.561674 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.561677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.561763 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.562492 (+ 729us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.562627 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3943,"mutex_wait_us":213,"spinlock_wait_cycles":50432,"thread_start_us":1043,"threads_started":1}
I20260808 04:45:32.563578 16675 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33995
I20260808 04:45:32.563822 16675 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:32.564112 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.546320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:48703 (local address 127.4.217.60:44505)
0808 04:45:32.546456 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.546461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.546935 (+ 474us) server_negotiation.cc:407] Connection header received
0808 04:45:32.547699 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.547703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.547761 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.547816 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.549998 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.550525 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.551746 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.551990 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.552108 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.555782 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.555786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.555791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.555825 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.557692 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.558526 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.558531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.558535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.558635 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.559128 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.559133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.559136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.559445 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.559545 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.563695 (+ 4150us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.563865 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:45:32.564404 16675 heartbeater.cc:511] Master 127.4.217.62:33995 requested a full tablet report, sending...
I20260808 04:45:32.564821 16676 heartbeater.cc:347] Connected to a master server at 127.4.217.60:44505
I20260808 04:45:32.564925 16676 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:32.564836 16678 heartbeater.cc:347] Connected to a master server at 127.4.217.61:37733
I20260808 04:45:32.565164 16676 heartbeater.cc:511] Master 127.4.217.60:44505 requested a full tablet report, sending...
I20260808 04:45:32.565172 16678 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:32.565618 16678 heartbeater.cc:511] Master 127.4.217.61:37733 requested a full tablet report, sending...
I20260808 04:45:32.565814 14862 ts_manager.cc:194] Re-registered known tserver with Master: 51ab903a54744f3eb3b323ad5a297485 (127.4.217.2:43665)
I20260808 04:45:32.566037 16150 ts_manager.cc:194] Registered new tserver with Master: 51ab903a54744f3eb3b323ad5a297485 (127.4.217.2:43665)
I20260808 04:45:32.566452 14862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:43111
I20260808 04:45:32.566900 15849 ts_manager.cc:194] Registered new tserver with Master: 51ab903a54744f3eb3b323ad5a297485 (127.4.217.2:43665)
I20260808 04:45:32.631151 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.627730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.0.0.1:42658)
0808 04:45:32.627868 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.627887 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.628003 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.628316 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.628320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.628339 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:32.628442 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.628448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.629282 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.629286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.630021 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.630030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.630728 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.630732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.630818 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.630849 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.630921 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.631003 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 04:45:32.631151 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.627827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.4.217.60:44505)
0808 04:45:32.627980 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.627985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.628002 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:45:32.628137 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.628140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.628203 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.628300 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:32.628569 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.629161 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.630158 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.630597 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.630719 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.630896 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.630940 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.631003 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:45:32.631814 16171 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 04:45:32.632202 16219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27cbf01bd965461fba0d841fd3a2ef81. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.632359 16219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: This master's current role is: LEARNER
W20260808 04:45:32.676546 16688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:32.676848 16688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:32.676955 16688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:32.680936 16688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:32.681011 16688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:32.681103 16688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:32.686215 16688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab
--principal=oryx/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:34263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=38905
--tserver_master_addrs=127.4.217.62:33995,127.4.217.61:37733,127.4.217.60:44505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:32.687247 16688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:32.688287 16688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:32.698361 16694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:32.698379 16693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:32.698377 16696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:32.699115 16688 server_base.cc:1034] running on GCE node
I20260808 04:45:32.699559 16688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:32.700208 16688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:32.701395 16688 hybrid_clock.cc:648] HybridClock initialized: now 1786164332701370 us; error 40 us; skew 500 ppm
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:32.704047 16688 init.cc:377] Logged in from keytab as oryx/127.4.217.3@KRBTEST.COM (short username oryx)
I20260808 04:45:32.704952 16688 webserver.cc:460] Webserver started at http://127.4.217.3:38905/ using document root <none> and password file <none>
I20260808 04:45:32.705493 16688 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:32.705590 16688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:32.709403 16688 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 04:45:32.711441 16707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.712682 16688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:32.712838 16688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "91bc72a32d1c4a729cfee1ebbaeba917"
format_stamp: "Formatted at 2026-08-08 04:45:31 on dist-test-slave-2kcd"
server_key: "69b6a8fd8663c57485c52937e66202a6"
server_key_iv: "8bc60d836c4ff92af475a943fe34b3b9"
server_key_version: "encryptionkey@0"
I20260808 04:45:32.713486 16688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:32.741169 16688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:32.744231 16688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:32.744397 16688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:32.745023 16688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:32.745903 16688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:32.745961 16688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.746006 16688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:32.746029 16688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:32.755594 16688 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:34263
I20260808 04:45:32.755672 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:34263 every 8 connection(s)
I20260808 04:45:32.756672 16688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:32.764747 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 16688
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.61@KRBTEST.COM
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
Aug 08 04:45:32 dist-test-slave-2kcd krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164332, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.60@KRBTEST.COM
I20260808 04:45:32.776576 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.759239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.3:60895)
0808 04:45:32.759791 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.759842 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.761092 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.761636 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.761643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.762079 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.762554 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.762561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.764306 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.764311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.765422 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.765462 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.766286 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.766290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.766448 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.767277 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.767293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.770126 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.772905 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.772912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.772917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.773616 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.773939 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.773942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.773944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.774129 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.774596 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.774602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.774908 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.775115 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.775469 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":252,"spinlock_wait_cycles":30208,"thread_start_us":125,"threads_started":1}
I20260808 04:45:32.777386 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.758616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:60895 (local address 127.4.217.62:33995)
0808 04:45:32.758758 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.758765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.760099 (+ 1334us) server_negotiation.cc:407] Connection header received
0808 04:45:32.761322 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.761326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.761383 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.761437 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.763641 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.764188 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.765647 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.766032 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.766119 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.770356 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.770359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.770362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.770387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.772736 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.773729 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.773732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.773734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.773832 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.774249 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.774253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.774255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.774455 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.774541 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.777076 (+ 2535us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.777204 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:45:32.777626 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.758721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:59991 (local address 127.4.217.60:44505)
0808 04:45:32.758844 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.758847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.759946 (+ 1099us) server_negotiation.cc:407] Connection header received
0808 04:45:32.761294 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.761297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.761363 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.761459 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.762681 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.763500 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.765449 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.765787 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.765865 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.773020 (+ 7155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.773025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.773031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.773063 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.775084 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.776071 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.776078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.776081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.776177 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.776574 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.776579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.776581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.776843 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.776932 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.777320 (+ 388us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.777440 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:45:32.778164 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.759239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.4.217.3:59991)
0808 04:45:32.759791 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.759843 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.761091 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.761635 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.761642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.762037 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.762548 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.762557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.763653 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.763656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.765303 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.765314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.766073 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.766077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.766393 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.767190 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.767217 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.772892 (+ 5675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.775226 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.775231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.775236 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.775551 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.776322 (+ 771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.776327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.776330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.776430 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.776999 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.777002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.777105 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.777741 (+ 636us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.777904 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":263,"spinlock_wait_cycles":19328,"thread_start_us":194,"threads_started":1}
I20260808 04:45:32.778363 16900 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33995
I20260808 04:45:32.778613 16900 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:32.779129 16900 heartbeater.cc:511] Master 127.4.217.62:33995 requested a full tablet report, sending...
I20260808 04:45:32.779443 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.760330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.4.217.3:41101)
0808 04:45:32.760738 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.760753 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.761092 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.761636 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.761643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.762079 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:32.762537 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.762557 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.763653 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.763656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.765302 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.765314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.766073 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.766077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.766404 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.767191 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:32.767211 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:32.770148 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:32.772413 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.772418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.772434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.778006 (+ 5572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.778335 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:32.778339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:32.778341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:32.778434 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:32.778886 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:32.778890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:32.778995 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.779089 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.779226 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":329,"spinlock_wait_cycles":1284096,"thread_start_us":167,"threads_started":1}
I20260808 04:45:32.779698 16905 heartbeater.cc:347] Connected to a master server at 127.4.217.60:44505
I20260808 04:45:32.779775 16905 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:32.779924 16905 heartbeater.cc:511] Master 127.4.217.60:44505 requested a full tablet report, sending...
I20260808 04:45:32.780771 16150 ts_manager.cc:194] Registered new tserver with Master: 91bc72a32d1c4a729cfee1ebbaeba917 (127.4.217.3:34263)
I20260808 04:45:32.780813 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.760448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:41101 (local address 127.4.217.61:37733)
0808 04:45:32.760583 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.760588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.760841 (+ 253us) server_negotiation.cc:407] Connection header received
0808 04:45:32.761322 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.761326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.761383 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.761437 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.762725 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.763500 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.765449 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.765793 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.765865 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.770318 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.770321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.770324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.770351 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.772276 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.778132 (+ 5856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.778136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.778138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.778224 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.778544 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.778547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.778549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.778758 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.780407 (+ 1649us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.780494 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.780613 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:45:32.781316 14862 ts_manager.cc:194] Re-registered known tserver with Master: 91bc72a32d1c4a729cfee1ebbaeba917 (127.4.217.3:34263)
I20260808 04:45:32.781436 16906 heartbeater.cc:347] Connected to a master server at 127.4.217.61:37733
I20260808 04:45:32.781517 16906 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:32.781664 16906 heartbeater.cc:511] Master 127.4.217.61:37733 requested a full tablet report, sending...
I20260808 04:45:32.781704 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.767880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.4.217.62:33995)
0808 04:45:32.768891 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.768913 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.768926 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:32.768963 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.768966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.769010 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.769098 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.770836 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.771664 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.772932 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.773664 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.774210 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.775451 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.775455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.775457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.775488 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.779595 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.780056 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.780059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.780061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.780149 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.780455 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.780458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.780459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.780675 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.781315 (+ 640us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.781401 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.781551 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":65,"threads_started":1}
I20260808 04:45:32.782488 14862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:60895
I20260808 04:45:32.782616 15849 ts_manager.cc:194] Registered new tserver with Master: 91bc72a32d1c4a729cfee1ebbaeba917 (127.4.217.3:34263)
I20260808 04:45:32.785923 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.768451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.4.217.61:37733)
0808 04:45:32.770204 (+ 1753us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.770210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.771700 (+ 1490us) server_negotiation.cc:407] Connection header received
0808 04:45:32.771750 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.771755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.771808 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.771903 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.772978 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.773759 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.774841 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.775171 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.775257 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.777190 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.777194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.777197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.777221 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.784055 (+ 6834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.784536 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.784539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.784541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.784638 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.784956 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.784963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.784965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.785212 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.785368 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.785622 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.785748 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1673,"thread_start_us":196,"threads_started":1}
I20260808 04:45:32.785848 14862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:45:32.789347 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.769521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.4.217.60:44505)
0808 04:45:32.775797 (+ 6276us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.775802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.775815 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:32.775851 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.775854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.775895 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.780412 (+ 4517us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.780509 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.781186 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.782385 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.782700 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.784373 (+ 1673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.785921 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.785924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.785927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.785951 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.787594 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.787988 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.787991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.787993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.788080 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.788335 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.788338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.788339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.788569 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.788700 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.788948 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.789087 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6202,"thread_start_us":77,"threads_started":1}
W20260808 04:45:32.790128 14862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:32.792663 15779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: NON_VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: true } } }
I20260808 04:45:32.792827 15779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:32.795990 15779 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: attempting to promote NON_VOTER 5d047bb613c04b4697dc7bbacc315424 to VOTER
I20260808 04:45:32.796305 15779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } }
I20260808 04:45:32.798252 16169 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 LEARNER]: Refusing update from remote peer 27cbf01bd965461fba0d841fd3a2ef81: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 04:45:32.798226 15862 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Refusing update from remote peer 27cbf01bd965461fba0d841fd3a2ef81: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 04:45:32.798799 16932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 04:45:32.799140 16932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 04:45:32.801039 16932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5d047bb613c04b4697dc7bbacc315424 (127.4.217.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:32.802215 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:32.802322 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:32.802450 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:32.802525 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:32.803318 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:32.803436 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:32.803447 16169 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5d047bb613c04b4697dc7bbacc315424 (127.4.217.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:32.803740 15862 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5d047bb613c04b4697dc7bbacc315424 (127.4.217.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:32.805289 16219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:32.805414 16219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:45:32.805621 16949 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27cbf01bd965461fba0d841fd3a2ef81" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:32.805860 16949 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:45:32.813136 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.807528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:36669 (local address 127.0.0.1:44316)
0808 04:45:32.807699 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.807714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.807796 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.808695 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.808698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.808720 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:32.808881 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.808890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.810491 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.810495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.811581 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.811594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.812679 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.812683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.812766 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.812814 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.812884 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.812994 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":26}
I20260808 04:45:32.813166 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.807694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.4.217.1:36669)
0808 04:45:32.808103 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.808122 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.808149 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:45:32.808243 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.808249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.808492 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.809049 (+ 557us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:32.809202 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.810356 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.811732 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.812556 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.812677 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.812894 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.812952 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.813045 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":202,"threads_started":1}
I20260808 04:45:32.813534 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.807837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:34263 (local address 127.0.0.1:53936)
0808 04:45:32.808918 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.808932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.809028 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.809473 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.809476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.809494 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:32.809613 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.809620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.810682 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.810687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.811435 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.811443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.812581 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.812584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.812729 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.812756 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.812850 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.813401 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":435,"thread_start_us":98,"threads_started":1}
I20260808 04:45:32.813786 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.808281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.4.217.3:34263)
0808 04:45:32.808726 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.808738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.809138 (+ 400us) server_negotiation.cc:407] Connection header received
0808 04:45:32.809206 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.809209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.809339 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.809497 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:32.809722 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.810580 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.811556 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.812265 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.812369 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.812515 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.812998 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.813103 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":104,"threads_started":1}
I20260808 04:45:32.815310 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.808295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:43665 (local address 127.0.0.1:47470)
0808 04:45:32.809514 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.809528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.809622 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.810099 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.810102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.810122 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:32.810207 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.810213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.812147 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.812150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.813484 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.813496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.814869 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.814876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.814969 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.814994 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.815069 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.815180 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"thread_start_us":106,"threads_started":1}
I20260808 04:45:32.815704 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.808740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.4.217.2:43665)
0808 04:45:32.809380 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.809390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.809714 (+ 324us) server_negotiation.cc:407] Connection header received
0808 04:45:32.809789 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.809794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.809957 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.810104 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:32.810326 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.812013 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.813627 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.814527 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.814667 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.814802 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.815488 (+ 686us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.815595 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":120,"threads_started":1}
I20260808 04:45:32.816874 16330 tablet_service.cc:1467] Processing CreateTablet for tablet f2c4f25d5c80461795bedaecf2844fee (DEFAULT_TABLE table=test-table [id=cfc87143bbc74b388e785bed877c6d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:32.816860 16781 tablet_service.cc:1467] Processing CreateTablet for tablet f2c4f25d5c80461795bedaecf2844fee (DEFAULT_TABLE table=test-table [id=cfc87143bbc74b388e785bed877c6d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:32.818044 16781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c4f25d5c80461795bedaecf2844fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:32.818055 16330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c4f25d5c80461795bedaecf2844fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:32.818256 16561 tablet_service.cc:1467] Processing CreateTablet for tablet f2c4f25d5c80461795bedaecf2844fee (DEFAULT_TABLE table=test-table [id=cfc87143bbc74b388e785bed877c6d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:32.819366 16561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c4f25d5c80461795bedaecf2844fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:32.825902 16960 tablet_bootstrap.cc:492] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: Bootstrap starting.
I20260808 04:45:32.827183 16961 tablet_bootstrap.cc:492] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: Bootstrap starting.
I20260808 04:45:32.828105 16960 tablet_bootstrap.cc:654] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:32.828933 16960 log.cc:824] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:32.829016 16961 tablet_bootstrap.cc:654] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:32.829130 16963 tablet_bootstrap.cc:492] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: Bootstrap starting.
I20260808 04:45:32.829895 16961 log.cc:824] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:32.831048 16960 tablet_bootstrap.cc:492] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: No bootstrap required, opened a new log
I20260808 04:45:32.831192 16963 tablet_bootstrap.cc:654] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:32.831245 16960 ts_tablet_manager.cc:1397] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 04:45:32.831873 16963 log.cc:824] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:32.831980 16961 tablet_bootstrap.cc:492] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: No bootstrap required, opened a new log
I20260808 04:45:32.832162 16961 ts_tablet_manager.cc:1397] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:45:32.833559 16963 tablet_bootstrap.cc:492] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: No bootstrap required, opened a new log
I20260808 04:45:32.833755 16963 ts_tablet_manager.cc:1397] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:45:32.836843 16960 raft_consensus.cc:348] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } }
I20260808 04:45:32.837112 16960 raft_consensus.cc:374] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:32.836985 16961 raft_consensus.cc:348] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } }
I20260808 04:45:32.837169 16960 raft_consensus.cc:729] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ab903a54744f3eb3b323ad5a297485, State: Initialized, Role: FOLLOWER
I20260808 04:45:32.837198 16961 raft_consensus.cc:374] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:32.837242 16961 raft_consensus.cc:729] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e274a0492549cb8762a91403cff09d, State: Initialized, Role: FOLLOWER
I20260808 04:45:32.837661 16960 consensus_queue.cc:260] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } }
I20260808 04:45:32.837661 16961 consensus_queue.cc:260] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } }
I20260808 04:45:32.837894 16963 raft_consensus.cc:348] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } }
I20260808 04:45:32.838111 16963 raft_consensus.cc:374] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:32.838153 16963 raft_consensus.cc:729] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bc72a32d1c4a729cfee1ebbaeba917, State: Initialized, Role: FOLLOWER
I20260808 04:45:32.838586 16963 consensus_queue.cc:260] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } }
I20260808 04:45:32.838748 16960 ts_tablet_manager.cc:1428] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:45:32.838748 16961 ts_tablet_manager.cc:1428] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 04:45:32.839140 16675 heartbeater.cc:503] Master 127.4.217.62:33995 was elected leader, sending a full tablet report...
I20260808 04:45:32.839600 16963 ts_tablet_manager.cc:1428] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:45:32.839972 16900 heartbeater.cc:503] Master 127.4.217.62:33995 was elected leader, sending a full tablet report...
I20260808 04:45:32.840273 16436 heartbeater.cc:503] Master 127.4.217.62:33995 was elected leader, sending a full tablet report...
W20260808 04:45:32.855298 16441 tablet.cc:2369] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:32.873900 16973 raft_consensus.cc:482] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:32.874110 16973 raft_consensus.cc:504] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } }
I20260808 04:45:32.875257 16973 leader_election.cc:288] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e274a0492549cb8762a91403cff09d (127.4.217.1:36669), 91bc72a32d1c4a729cfee1ebbaeba917 (127.4.217.3:34263)
I20260808 04:45:32.879106 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.875642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:38047 (local address 127.4.217.3:34263)
0808 04:45:32.875787 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.875792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.875807 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:32.875934 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.875937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.875988 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.876076 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:32.876394 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.876941 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.878093 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.878493 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.878614 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.878680 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.878873 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.878954 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:45:32.879112 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.875627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:48955 (local address 127.4.217.1:36669)
0808 04:45:32.875758 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.875763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.875778 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:32.875936 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.875938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.875983 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.876072 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:32.876386 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.876942 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.878117 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.878530 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.878648 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.878712 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.878882 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.878972 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:45:32.879127 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.875533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:34263 (local address 127.4.217.2:38047)
0808 04:45:32.875729 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.875743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.875829 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.876088 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.876091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.876117 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:32.876277 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.876282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.877052 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.877055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.877959 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.877968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.878629 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.878633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.878737 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.878767 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.878853 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.878968 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":127,"spinlock_wait_cycles":768}
I20260808 04:45:32.879141 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.875456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:36669 (local address 127.4.217.2:48955)
0808 04:45:32.875675 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:32.875695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:32.875793 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:32.876083 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:32.876086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:32.876113 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:32.876272 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.876278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.877057 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.877062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.877963 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:32.877979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.878650 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.878654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:32.878754 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.878778 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:45:32.878863 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:45:32.879002 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":91,"spinlock_wait_cycles":1792}
I20260808 04:45:32.879873 16821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c4f25d5c80461795bedaecf2844fee" candidate_uuid: "51ab903a54744f3eb3b323ad5a297485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" is_pre_election: true
I20260808 04:45:32.880245 16821 raft_consensus.cc:2454] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ab903a54744f3eb3b323ad5a297485 in term 0.
I20260808 04:45:32.880254 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c4f25d5c80461795bedaecf2844fee" candidate_uuid: "51ab903a54744f3eb3b323ad5a297485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e274a0492549cb8762a91403cff09d" is_pre_election: true
I20260808 04:45:32.880563 16367 raft_consensus.cc:2454] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ab903a54744f3eb3b323ad5a297485 in term 0.
I20260808 04:45:32.880832 16482 leader_election.cc:302] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ab903a54744f3eb3b323ad5a297485, 91bc72a32d1c4a729cfee1ebbaeba917; no voters:
I20260808 04:45:32.881160 16973 raft_consensus.cc:2781] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:32.881266 16973 raft_consensus.cc:482] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:32.881325 16973 raft_consensus.cc:3037] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:32.882403 16973 raft_consensus.cc:504] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } }
I20260808 04:45:32.882799 16973 leader_election.cc:288] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [CANDIDATE]: Term 1 election: Requested vote from peers 54e274a0492549cb8762a91403cff09d (127.4.217.1:36669), 91bc72a32d1c4a729cfee1ebbaeba917 (127.4.217.3:34263)
I20260808 04:45:32.883127 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c4f25d5c80461795bedaecf2844fee" candidate_uuid: "51ab903a54744f3eb3b323ad5a297485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e274a0492549cb8762a91403cff09d"
I20260808 04:45:32.883240 16821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c4f25d5c80461795bedaecf2844fee" candidate_uuid: "51ab903a54744f3eb3b323ad5a297485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91bc72a32d1c4a729cfee1ebbaeba917"
I20260808 04:45:32.883272 16367 raft_consensus.cc:3037] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:32.883416 16821 raft_consensus.cc:3037] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:32.884434 16367 raft_consensus.cc:2454] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ab903a54744f3eb3b323ad5a297485 in term 1.
I20260808 04:45:32.884588 16821 raft_consensus.cc:2454] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ab903a54744f3eb3b323ad5a297485 in term 1.
I20260808 04:45:32.884766 16485 leader_election.cc:302] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ab903a54744f3eb3b323ad5a297485, 54e274a0492549cb8762a91403cff09d; no voters:
I20260808 04:45:32.885165 16973 raft_consensus.cc:2781] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:32.885406 16973 raft_consensus.cc:686] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 1 LEADER]: Becoming Leader. State: Replica: 51ab903a54744f3eb3b323ad5a297485, State: Running, Role: LEADER
I20260808 04:45:32.885768 16973 consensus_queue.cc:237] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } }
I20260808 04:45:32.889353 14860 catalog_manager.cc:5495] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51ab903a54744f3eb3b323ad5a297485 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "51ab903a54744f3eb3b323ad5a297485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } health_report { overall_health: HEALTHY } } }
I20260808 04:45:32.911952 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.908679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.4.217.2:43665)
0808 04:45:32.908854 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.908858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.908873 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:32.909017 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.909019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.909078 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.909166 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:32.909388 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.909866 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.910687 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.910891 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.910970 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.911235 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:32.911635 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:32.911725 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.911777 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.911830 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:45:32.918196 16821 raft_consensus.cc:1261] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [term 1 FOLLOWER]: Refusing update from remote peer 51ab903a54744f3eb3b323ad5a297485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:32.918200 16367 raft_consensus.cc:1261] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [term 1 FOLLOWER]: Refusing update from remote peer 51ab903a54744f3eb3b323ad5a297485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:32.918998 16973 consensus_queue.cc:1035] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:32.919220 16975 consensus_queue.cc:1035] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:32.929949 16980 mvcc.cc:205] Tried to move back new op lower bound from 7316129107625652224 to 7316129107503091712. Current Snapshot: MvccSnapshot[applied={T|T < 7316129107625652224}]
I20260808 04:45:32.939220 14862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:32.939433 14862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:32.944650 14862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27cbf01bd965461fba0d841fd3a2ef81: Sending DeleteTablet for 3 replicas of tablet f2c4f25d5c80461795bedaecf2844fee
I20260808 04:45:32.945406 16330 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c4f25d5c80461795bedaecf2844fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:32 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:44316
I20260808 04:45:32.945461 16781 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c4f25d5c80461795bedaecf2844fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:32 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:53936
I20260808 04:45:32.945989 16993 tablet_replica.cc:330] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: stopping tablet replica
I20260808 04:45:32.946281 16989 tablet_replica.cc:330] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: stopping tablet replica
I20260808 04:45:32.946266 16561 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c4f25d5c80461795bedaecf2844fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:32 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:47470
I20260808 04:45:32.946729 16989 raft_consensus.cc:2229] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:32.946764 16995 tablet_replica.cc:330] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: stopping tablet replica
I20260808 04:45:32.947090 16995 raft_consensus.cc:2229] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:32.947119 16993 raft_consensus.cc:2229] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:32.947536 16995 raft_consensus.cc:2258] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:32.947553 16993 raft_consensus.cc:2258] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:32.947909 16989 raft_consensus.cc:2258] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:32.949043 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 14811
I20260808 04:45:32.957481 16995 ts_tablet_manager.cc:1905] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:32.960549 16993 ts_tablet_manager.cc:1905] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:32.961624 16995 ts_tablet_manager.cc:1918] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:32.961635 16989 ts_tablet_manager.cc:1905] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:32.961740 16995 log.cc:1196] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/f2c4f25d5c80461795bedaecf2844fee
I20260808 04:45:32.962119 16995 ts_tablet_manager.cc:1939] T f2c4f25d5c80461795bedaecf2844fee P 51ab903a54744f3eb3b323ad5a297485: Deleting consensus metadata
I20260808 04:45:32.967276 16989 ts_tablet_manager.cc:1918] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:32.967425 16993 ts_tablet_manager.cc:1918] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:32.967587 16989 log.cc:1196] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/f2c4f25d5c80461795bedaecf2844fee
I20260808 04:45:32.967706 16993 log.cc:1196] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/f2c4f25d5c80461795bedaecf2844fee
I20260808 04:45:32.968108 16989 ts_tablet_manager.cc:1939] T f2c4f25d5c80461795bedaecf2844fee P 54e274a0492549cb8762a91403cff09d: Deleting consensus metadata
W20260808 04:45:32.968281 16485 connection.cc:535] server connection from 127.0.0.1:47470 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:45:32.968577 16993 ts_tablet_manager.cc:1939] T f2c4f25d5c80461795bedaecf2844fee P 91bc72a32d1c4a729cfee1ebbaeba917: Deleting consensus metadata
W20260808 04:45:32.970489 16256 connection.cc:441] server connection from 127.0.0.1:44316 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44316 (request call id 1) could send its response
W20260808 04:45:32.971361 16710 connection.cc:441] server connection from 127.0.0.1:53936 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53936 (request call id 1) could send its response
I20260808 04:45:32.979514 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.948260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.4.217.61:37733)
0808 04:45:32.948497 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.948502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.969779 (+ 21277us) server_negotiation.cc:407] Connection header received
0808 04:45:32.969948 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.969954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.970025 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.970115 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.971191 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.971916 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.973058 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.973414 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.973500 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.975469 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.975473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.975476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.975505 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.977262 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.977746 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.977750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.977752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.977842 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.978181 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.978185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.978187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.978388 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.978506 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.979186 (+ 680us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.979334 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260808 04:45:32.983908 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.972464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.4.217.61:37733)
0808 04:45:32.972584 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.972588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.972917 (+ 329us) server_negotiation.cc:407] Connection header received
0808 04:45:32.973087 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.973090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.973133 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.973179 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.974740 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.975521 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.976522 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.976754 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.976870 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.978759 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.978762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.978765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.978791 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.981273 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.982345 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.982350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.982354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.982451 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.982812 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.982818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.982819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.983047 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.983153 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.983578 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.983705 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:45:32.986737 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.973027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.4.217.60:44505)
0808 04:45:32.973141 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.973145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.977079 (+ 3934us) server_negotiation.cc:407] Connection header received
0808 04:45:32.977500 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.977507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.977581 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.977644 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.978582 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.979410 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.980856 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.981259 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.981350 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.983009 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.983012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.983015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.983053 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.984869 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.985372 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.985376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.985377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.985468 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.985785 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.985788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.985790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.986021 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.986159 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.986446 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.986564 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260808 04:45:32.988867 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:32.948374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.4.217.60:44505)
0808 04:45:32.948488 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:32.948494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:32.979079 (+ 30585us) server_negotiation.cc:407] Connection header received
0808 04:45:32.980061 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:32.980064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:32.980116 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:32.980164 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:32.980959 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.981769 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.982944 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:32.983302 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:32.983372 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:32.985541 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:32.985548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:32.985551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:32.985578 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:32.987117 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.987595 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.987599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.987601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.987692 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:32.987976 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:32.987980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:32.987981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:32.988149 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:32.988277 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:32.988520 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 04:45:32.988660 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":31}
I20260808 04:45:33.931994 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:33.931539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.3:41969)
0808 04:45:33.931807 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:33.931892 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
W20260808 04:45:33.932490 16900 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:33995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
I20260808 04:45:33.933985 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:33.933547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.2:35715)
0808 04:45:33.933787 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:33.933884 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
W20260808 04:45:33.934304 16675 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:33995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
I20260808 04:45:33.935319 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:33.934747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.1:36919)
0808 04:45:33.935105 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:33.935199 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":64,"threads_started":1}
W20260808 04:45:33.935621 16436 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:33995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
I20260808 04:45:34.444207 17015 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27cbf01bd965461fba0d841fd3a2ef81)
I20260808 04:45:34.444490 17015 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } }
I20260808 04:45:34.444983 17017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27cbf01bd965461fba0d841fd3a2ef81)
I20260808 04:45:34.445200 17017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } }
I20260808 04:45:34.445864 17015 leader_election.cc:288] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27cbf01bd965461fba0d841fd3a2ef81 (127.4.217.62:33995), 5d047bb613c04b4697dc7bbacc315424 (127.4.217.60:44505)
I20260808 04:45:34.446408 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.445940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:45192)
0808 04:45:34.446192 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.446297 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
I20260808 04:45:34.446393 17017 leader_election.cc:288] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27cbf01bd965461fba0d841fd3a2ef81 (127.4.217.62:33995), a4c3332e4bd74a3bb886772254bc27d1 (127.4.217.61:37733)
I20260808 04:45:34.446514 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.446103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:45196)
0808 04:45:34.446338 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.446410 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
W20260808 04:45:34.446704 16130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
W20260808 04:45:34.446870 15827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
I20260808 04:45:34.447834 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.447549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:45214)
0808 04:45:34.447695 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.447740 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260808 04:45:34.447948 15827 leader_election.cc:334] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27cbf01bd965461fba0d841fd3a2ef81 (127.4.217.62:33995): Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
I20260808 04:45:34.448061 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.447516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:45198)
0808 04:45:34.447925 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.447981 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341}
W20260808 04:45:34.448181 16130 leader_election.cc:334] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27cbf01bd965461fba0d841fd3a2ef81 (127.4.217.62:33995): Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
I20260808 04:45:34.450650 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.446460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.4.217.60:44505)
0808 04:45:34.446941 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:34.446947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:34.446960 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:34.447006 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:34.447009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:34.447078 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:34.447496 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:34.447603 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.448427 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.449574 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.450150 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.450261 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.450340 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:34.450490 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 04:45:34.450562 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":99,"threads_started":1}
I20260808 04:45:34.450764 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.446077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:44505 (local address 127.0.0.1:42694)
0808 04:45:34.446622 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.446639 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:34.446733 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:34.447220 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:34.447224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:34.447245 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:34.447376 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.447384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.448539 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.448543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.449404 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.449413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.450263 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.450266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.450360 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.450390 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:45:34.450468 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:45:34.450597 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":29,"thread_start_us":73,"threads_started":1}
I20260808 04:45:34.451280 16169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c3332e4bd74a3bb886772254bc27d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5d047bb613c04b4697dc7bbacc315424" is_pre_election: true
I20260808 04:45:34.451509 16169 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4c3332e4bd74a3bb886772254bc27d1 in term 1.
I20260808 04:45:34.451618 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.446967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.4.217.61:37733)
0808 04:45:34.447609 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:34.447619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:34.447631 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:45:34.447666 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:34.447669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:34.447707 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:34.447791 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:34.448343 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.448954 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.450381 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.451072 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.451167 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.451208 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:34.451430 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 04:45:34.451510 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":575,"thread_start_us":115,"threads_started":1}
I20260808 04:45:34.451638 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.446736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:37733 (local address 127.0.0.1:51546)
0808 04:45:34.447169 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.447183 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:34.447277 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:34.447829 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:34.447832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:34.447857 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:34.448185 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.448200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.449066 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.449069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.450264 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.450272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.451192 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.451195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.451279 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.451312 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:45:34.451399 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:45:34.451473 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":63,"threads_started":1}
I20260808 04:45:34.451879 15826 leader_election.cc:302] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d047bb613c04b4697dc7bbacc315424, a4c3332e4bd74a3bb886772254bc27d1; no voters: 27cbf01bd965461fba0d841fd3a2ef81
I20260808 04:45:34.452167 17015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 04:45:34.452241 17015 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27cbf01bd965461fba0d841fd3a2ef81)
I20260808 04:45:34.452304 17015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:45:34.452466 15862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d047bb613c04b4697dc7bbacc315424" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a4c3332e4bd74a3bb886772254bc27d1" is_pre_election: true
I20260808 04:45:34.453595 17015 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } }
I20260808 04:45:34.453857 15862 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d047bb613c04b4697dc7bbacc315424 in current term 2: Already voted for candidate a4c3332e4bd74a3bb886772254bc27d1 in this term.
I20260808 04:45:34.454070 17015 leader_election.cc:288] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [CANDIDATE]: Term 2 election: Requested vote from peers 27cbf01bd965461fba0d841fd3a2ef81 (127.4.217.62:33995), 5d047bb613c04b4697dc7bbacc315424 (127.4.217.60:44505)
I20260808 04:45:34.454411 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.454163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:45228)
0808 04:45:34.454255 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.454321 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 04:45:34.454487 16169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c3332e4bd74a3bb886772254bc27d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5d047bb613c04b4697dc7bbacc315424"
I20260808 04:45:34.454623 16169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:45:34.454828 16128 leader_election.cc:302] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d047bb613c04b4697dc7bbacc315424; no voters: 27cbf01bd965461fba0d841fd3a2ef81, a4c3332e4bd74a3bb886772254bc27d1
I20260808 04:45:34.455773 16169 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4c3332e4bd74a3bb886772254bc27d1 in term 2.
I20260808 04:45:34.455905 17017 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:45:34.456156 15826 leader_election.cc:302] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d047bb613c04b4697dc7bbacc315424, a4c3332e4bd74a3bb886772254bc27d1; no voters:
I20260808 04:45:34.456350 17015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:45:34.456595 17015 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [term 2 LEADER]: Becoming Leader. State: Replica: a4c3332e4bd74a3bb886772254bc27d1, State: Running, Role: LEADER
I20260808 04:45:34.456691 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.456420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:45244)
0808 04:45:34.456562 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.456630 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260808 04:45:34.456827 15827 leader_election.cc:334] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 27cbf01bd965461fba0d841fd3a2ef81 (127.4.217.62:33995): Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
I20260808 04:45:34.456921 17015 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } }
I20260808 04:45:34.458123 17029 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c3332e4bd74a3bb886772254bc27d1. Latest consensus state: current_term: 2 leader_uuid: "a4c3332e4bd74a3bb886772254bc27d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:34.458251 17029 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:34.458570 17032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:34.461100 17032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:34.461494 17032 catalog_manager.cc:1259] Loaded cluster ID: 546263cf6e8a449d960d1a054e92e70e
I20260808 04:45:34.461561 17032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:34.462113 17032 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:34.462507 17032 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Loaded TSK: 0
I20260808 04:45:34.463179 17032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:45:34.506596 15848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:34.509784 15848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:34.512771 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.512508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:45270)
0808 04:45:34.512632 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.512682 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 04:45:34.512826 16169 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [term 2 FOLLOWER]: Refusing update from remote peer a4c3332e4bd74a3bb886772254bc27d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 04:45:34.513381 17015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 04:45:34.513566 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.513309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.0.0.1:45272)
0808 04:45:34.513418 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.513482 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:45:34.513712 15827 consensus_peers.cc:487] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 -> Peer 27cbf01bd965461fba0d841fd3a2ef81 (127.4.217.62:33995): Couldn't send request to peer 27cbf01bd965461fba0d841fd3a2ef81. Status: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:45:34.515080 17017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c3332e4bd74a3bb886772254bc27d1. Latest consensus state: current_term: 2 leader_uuid: "a4c3332e4bd74a3bb886772254bc27d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:34.515218 17017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:45:34.516360 17015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a4c3332e4bd74a3bb886772254bc27d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:34.516490 17015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:34.516911 17017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a4c3332e4bd74a3bb886772254bc27d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27cbf01bd965461fba0d841fd3a2ef81" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33995 } } peers { permanent_uuid: "a4c3332e4bd74a3bb886772254bc27d1" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 37733 } attrs { promote: false } } peers { permanent_uuid: "5d047bb613c04b4697dc7bbacc315424" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 44505 } attrs { promote: false } } }
I20260808 04:45:34.517050 17017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d047bb613c04b4697dc7bbacc315424 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:45:34.528820 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.523874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:43665 (local address 127.0.0.1:47488)
0808 04:45:34.524244 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.524259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:34.524352 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:34.524899 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:34.524903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:34.524937 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:34.525102 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.525109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.526400 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.526403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.527536 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.527547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.528363 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.528367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.528455 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.528497 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 04:45:34.528548 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 04:45:34.528640 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":67,"spinlock_wait_cycles":8576}
I20260808 04:45:34.529201 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.524968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:36669 (local address 127.0.0.1:44318)
0808 04:45:34.525292 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.525302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:34.525369 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:34.525744 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:34.525746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:34.525763 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:34.525875 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.525880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.526727 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.526731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.527832 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.527842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.528698 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.528702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.528848 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.528879 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:45:34.528960 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 04:45:34.529025 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":85,"threads_started":1}
I20260808 04:45:34.529250 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.524019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.4.217.2:43665)
0808 04:45:34.524603 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:34.524608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:34.524624 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:45:34.524677 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:34.524680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:34.524729 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:34.525405 (+ 676us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:34.525548 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.526280 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.527680 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.528228 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.528899 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.528982 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:34.529024 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 04:45:34.529098 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":87,"threads_started":1}
I20260808 04:45:34.529332 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.524974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.4.217.1:36669)
0808 04:45:34.525545 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:34.525549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:34.525560 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:45:34.525603 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:34.525605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:34.525640 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:34.525736 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:34.525994 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.526608 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.527961 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.528591 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.528697 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.528757 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:34.529093 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 04:45:34.529180 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":79,"threads_started":1}
I20260808 04:45:34.529627 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.524293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.4.217.3:34263)
0808 04:45:34.524577 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:34.524583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:34.524601 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:45:34.524663 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:34.524666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:34.524721 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:34.525039 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:34.525403 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.526255 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.527872 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.528344 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.528479 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.528552 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:34.529364 (+ 812us) server_negotiation.cc:299] Negotiation successful
0808 04:45:34.529458 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
I20260808 04:45:34.529035 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.524194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:34263 (local address 127.0.0.1:53948)
0808 04:45:34.524282 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.524296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:34.524360 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:34.524902 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:34.524905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:34.524935 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:34.525275 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.525279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.526416 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.526420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.527677 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.527689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.528483 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.528487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.528596 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.528624 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:45:34.528703 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 04:45:34.528805 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":207,"spinlock_wait_cycles":5760}
I20260808 04:45:34.529855 16561 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5ea00775a845a2a67db9de58a9f305 (DEFAULT_TABLE table=test-table [id=72c84b48d6ac41e5aac3c1a94566eb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:34.530212 16561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5ea00775a845a2a67db9de58a9f305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:34.530390 16781 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5ea00775a845a2a67db9de58a9f305 (DEFAULT_TABLE table=test-table [id=72c84b48d6ac41e5aac3c1a94566eb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:34.530691 16781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5ea00775a845a2a67db9de58a9f305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:34.533165 16330 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5ea00775a845a2a67db9de58a9f305 (DEFAULT_TABLE table=test-table [id=72c84b48d6ac41e5aac3c1a94566eb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:34.533228 17047 tablet_bootstrap.cc:492] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: Bootstrap starting.
I20260808 04:45:34.533457 16330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5ea00775a845a2a67db9de58a9f305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:34.533767 17048 tablet_bootstrap.cc:492] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: Bootstrap starting.
I20260808 04:45:34.534226 17047 tablet_bootstrap.cc:654] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:34.534788 17048 tablet_bootstrap.cc:654] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:34.535948 17048 tablet_bootstrap.cc:492] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: No bootstrap required, opened a new log
I20260808 04:45:34.536007 17052 tablet_bootstrap.cc:492] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d: Bootstrap starting.
I20260808 04:45:34.536072 17048 ts_tablet_manager.cc:1397] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:34.536567 17048 raft_consensus.cc:348] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } }
I20260808 04:45:34.536700 17048 raft_consensus.cc:374] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:34.536737 17048 raft_consensus.cc:729] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bc72a32d1c4a729cfee1ebbaeba917, State: Initialized, Role: FOLLOWER
I20260808 04:45:34.536960 17052 tablet_bootstrap.cc:654] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:34.536969 17047 tablet_bootstrap.cc:492] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: No bootstrap required, opened a new log
I20260808 04:45:34.536929 17048 consensus_queue.cc:260] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } }
I20260808 04:45:34.537060 17047 ts_tablet_manager.cc:1397] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 04:45:34.537271 17048 ts_tablet_manager.cc:1428] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 04:45:34.537500 17047 raft_consensus.cc:348] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } }
I20260808 04:45:34.537616 17047 raft_consensus.cc:374] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:34.537648 17047 raft_consensus.cc:729] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ab903a54744f3eb3b323ad5a297485, State: Initialized, Role: FOLLOWER
I20260808 04:45:34.537763 17047 consensus_queue.cc:260] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } }
I20260808 04:45:34.538115 17047 ts_tablet_manager.cc:1428] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:34.538172 17052 tablet_bootstrap.cc:492] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d: No bootstrap required, opened a new log
I20260808 04:45:34.538287 17052 ts_tablet_manager.cc:1397] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 04:45:34.538825 17052 raft_consensus.cc:348] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } }
I20260808 04:45:34.538941 17052 raft_consensus.cc:374] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:34.538975 17052 raft_consensus.cc:729] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e274a0492549cb8762a91403cff09d, State: Initialized, Role: FOLLOWER
I20260808 04:45:34.539093 17052 consensus_queue.cc:260] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } }
I20260808 04:45:34.539392 17052 ts_tablet_manager.cc:1428] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:45:34.541075 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.537738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.3:37537)
0808 04:45:34.540877 (+ 3139us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.540951 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":95,"threads_started":1}
I20260808 04:45:34.541203 16906 heartbeater.cc:503] Master 127.4.217.61:37733 was elected leader, sending a full tablet report...
I20260808 04:45:34.541716 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.541491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.3:51009)
0808 04:45:34.541601 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.541654 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 04:45:34.542135 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.538578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.2:36795)
0808 04:45:34.541945 (+ 3367us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.542043 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2861,"thread_start_us":96,"threads_started":1}
W20260808 04:45:34.542135 16679 tablet.cc:2369] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:34.545997 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.545559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.1:33889)
0808 04:45:34.545824 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.545897 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260808 04:45:34.547181 16438 heartbeater.cc:503] Master 127.4.217.61:37733 was elected leader, sending a full tablet report...
I20260808 04:45:34.547533 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.547295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.1:47949)
0808 04:45:34.547395 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.547450 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 04:45:34.548147 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.547900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.2:44651)
0808 04:45:34.548014 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.548087 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 04:45:34.548509 16678 heartbeater.cc:503] Master 127.4.217.61:37733 was elected leader, sending a full tablet report...
I20260808 04:45:34.602193 17053 raft_consensus.cc:482] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:34.602387 17053 raft_consensus.cc:504] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } }
I20260808 04:45:34.603559 17053 leader_election.cc:288] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51ab903a54744f3eb3b323ad5a297485 (127.4.217.2:43665), 54e274a0492549cb8762a91403cff09d (127.4.217.1:36669)
I20260808 04:45:34.608615 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.604165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:52283 (local address 127.4.217.1:36669)
0808 04:45:34.604488 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:34.604494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:34.604862 (+ 368us) server_negotiation.cc:407] Connection header received
0808 04:45:34.605086 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:34.605092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:34.605158 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:34.605227 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:34.605684 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.606327 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.607482 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.607998 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.608081 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.608136 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:34.608374 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 04:45:34.608462 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260808 04:45:34.608629 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.603941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:36669 (local address 127.4.217.3:52283)
0808 04:45:34.604734 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.604752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:34.604949 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:34.605384 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:34.605386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:34.605402 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:34.605554 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.605562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.606440 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.606443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.607360 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.607370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.608138 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.608142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.608239 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.608267 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:45:34.608350 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:45:34.608432 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":34,"thread_start_us":203,"threads_started":1}
I20260808 04:45:34.608909 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.603941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:43665 (local address 127.4.217.3:53635)
0808 04:45:34.604176 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.604193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:34.604311 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:34.605826 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:34.605829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:34.605851 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:34.605956 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.605962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.606762 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.606765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.607694 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:34.607706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.608476 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.608480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:34.608578 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.608601 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:45:34.608687 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 04:45:34.608764 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260808 04:45:34.608940 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.605446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:53635 (local address 127.4.217.2:43665)
0808 04:45:34.605585 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:34.605591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:34.605608 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:45:34.605668 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:34.605672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:34.605725 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:34.605784 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:34.606070 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.606653 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.607821 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.608364 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.608461 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.608520 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:34.608722 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 04:45:34.608823 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 04:45:34.609351 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5ea00775a845a2a67db9de58a9f305" candidate_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e274a0492549cb8762a91403cff09d" is_pre_election: true
I20260808 04:45:34.609429 16595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5ea00775a845a2a67db9de58a9f305" candidate_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ab903a54744f3eb3b323ad5a297485" is_pre_election: true
I20260808 04:45:34.609530 16367 raft_consensus.cc:2454] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bc72a32d1c4a729cfee1ebbaeba917 in term 0.
I20260808 04:45:34.609699 16595 raft_consensus.cc:2454] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bc72a32d1c4a729cfee1ebbaeba917 in term 0.
I20260808 04:45:34.609973 16711 leader_election.cc:302] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e274a0492549cb8762a91403cff09d, 91bc72a32d1c4a729cfee1ebbaeba917; no voters:
I20260808 04:45:34.610265 17053 raft_consensus.cc:2781] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:34.610320 17053 raft_consensus.cc:482] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:34.610348 17053 raft_consensus.cc:3037] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:34.611361 17053 raft_consensus.cc:504] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } }
I20260808 04:45:34.611744 17053 leader_election.cc:288] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [CANDIDATE]: Term 1 election: Requested vote from peers 51ab903a54744f3eb3b323ad5a297485 (127.4.217.2:43665), 54e274a0492549cb8762a91403cff09d (127.4.217.1:36669)
I20260808 04:45:34.612119 16595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5ea00775a845a2a67db9de58a9f305" candidate_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ab903a54744f3eb3b323ad5a297485"
I20260808 04:45:34.612258 16595 raft_consensus.cc:3037] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:34.612327 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5ea00775a845a2a67db9de58a9f305" candidate_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e274a0492549cb8762a91403cff09d"
I20260808 04:45:34.612463 16367 raft_consensus.cc:3037] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:34.613485 16595 raft_consensus.cc:2454] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bc72a32d1c4a729cfee1ebbaeba917 in term 1.
I20260808 04:45:34.613605 16367 raft_consensus.cc:2454] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bc72a32d1c4a729cfee1ebbaeba917 in term 1.
I20260808 04:45:34.613786 16710 leader_election.cc:302] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ab903a54744f3eb3b323ad5a297485, 91bc72a32d1c4a729cfee1ebbaeba917; no voters:
I20260808 04:45:34.613966 17053 raft_consensus.cc:2781] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:34.614233 17053 raft_consensus.cc:686] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 1 LEADER]: Becoming Leader. State: Replica: 91bc72a32d1c4a729cfee1ebbaeba917, State: Running, Role: LEADER
I20260808 04:45:34.614501 17053 consensus_queue.cc:237] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } }
I20260808 04:45:34.615062 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.614822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.3:51253)
0808 04:45:34.614918 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.614998 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 04:45:34.616956 15848 catalog_manager.cc:5495] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91bc72a32d1c4a729cfee1ebbaeba917 (127.4.217.3). New cstate: current_term: 1 leader_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91bc72a32d1c4a729cfee1ebbaeba917" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 34263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:34.631815 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.628653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.4.217.3:34263)
0808 04:45:34.628819 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:34.628824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:34.628839 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:34.628918 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:34.628920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:34.628972 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:34.629059 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:34.629288 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.629802 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.630591 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:34.630794 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:34.630870 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:34.631084 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:34.631522 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:34.631607 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:34.631664 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 04:45:34.631710 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:45:34.634608 16367 raft_consensus.cc:1261] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d [term 1 FOLLOWER]: Refusing update from remote peer 91bc72a32d1c4a729cfee1ebbaeba917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:34.634863 16595 raft_consensus.cc:1261] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [term 1 FOLLOWER]: Refusing update from remote peer 91bc72a32d1c4a729cfee1ebbaeba917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:34.635131 17053 consensus_queue.cc:1035] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e274a0492549cb8762a91403cff09d" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:34.635442 17071 consensus_queue.cc:1035] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ab903a54744f3eb3b323ad5a297485" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:34.636705 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.636389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.1:50679)
0808 04:45:34.636560 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.636624 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 04:45:34.637554 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.637251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.2:49507)
0808 04:45:34.637424 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.637480 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260808 04:45:34.640039 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.639761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.1:33275)
0808 04:45:34.639926 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.639976 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260808 04:45:34.640574 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.640311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.3:60421)
0808 04:45:34.640436 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.640506 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:45:34.641454 17079 mvcc.cc:205] Tried to move back new op lower bound from 7316129114658406400 to 7316129114583789568. Current Snapshot: MvccSnapshot[applied={T|T < 7316129114658406400}]
I20260808 04:45:34.641940 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.641132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.2:52037)
0808 04:45:34.641808 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.641869 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596}
I20260808 04:45:34.642256 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.642022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33995 (local address 127.4.217.3:52889)
0808 04:45:34.642121 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.642172 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 04:45:34.651651 15848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:34.651834 15848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:34.656061 15848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4c3332e4bd74a3bb886772254bc27d1: Sending DeleteTablet for 3 replicas of tablet 6d5ea00775a845a2a67db9de58a9f305
I20260808 04:45:34.656770 16561 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5ea00775a845a2a67db9de58a9f305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:34 UTC) from {username='oryx', principal='oryx/127.4.217.61@KRBTEST.COM'} at 127.0.0.1:47488
I20260808 04:45:34.656937 16330 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5ea00775a845a2a67db9de58a9f305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:34 UTC) from {username='oryx', principal='oryx/127.4.217.61@KRBTEST.COM'} at 127.0.0.1:44318
I20260808 04:45:34.657203 16781 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5ea00775a845a2a67db9de58a9f305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:34 UTC) from {username='oryx', principal='oryx/127.4.217.61@KRBTEST.COM'} at 127.0.0.1:53948
I20260808 04:45:34.657539 17088 tablet_replica.cc:330] T 6d5ea00775a845a2a67db9de58a9f305 P 54e274a0492549cb8762a91403cff09d: stopping tablet replica
I20260808 04:45:34.657548 17087 tablet_replica.cc:330] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: stopping tablet replica
I20260808 04:45:34.657676 17087 raft_consensus.cc:2229] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:34.657819 17087 raft_consensus.cc:2258] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:34.657994 17089 tablet_replica.cc:330] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: stopping tablet replica
I20260808 04:45:34.658041 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 16095
I20260808 04:45:34.661607 17087 ts_tablet_manager.cc:1905] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:34.664680 17087 ts_tablet_manager.cc:1918] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:34.664757 17087 log.cc:1196] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/6d5ea00775a845a2a67db9de58a9f305
I20260808 04:45:34.665067 17087 ts_tablet_manager.cc:1939] T 6d5ea00775a845a2a67db9de58a9f305 P 51ab903a54744f3eb3b323ad5a297485: Deleting consensus metadata
I20260808 04:45:34.665769 15826 catalog_manager.cc:4841] TS 51ab903a54744f3eb3b323ad5a297485 (127.4.217.2:43665): tablet 6d5ea00775a845a2a67db9de58a9f305 (table test-table [id=72c84b48d6ac41e5aac3c1a94566eb29]) successfully deleted
I20260808 04:45:34.658198 17089 raft_consensus.cc:2229] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:34.669200 17089 raft_consensus.cc:2258] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:34.670254 17089 ts_tablet_manager.cc:1905] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:34.673333 17089 ts_tablet_manager.cc:1918] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:34.673439 17089 log.cc:1196] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/6d5ea00775a845a2a67db9de58a9f305
I20260808 04:45:34.673772 17089 ts_tablet_manager.cc:1939] T 6d5ea00775a845a2a67db9de58a9f305 P 91bc72a32d1c4a729cfee1ebbaeba917: Deleting consensus metadata
I20260808 04:45:34.677026 15823 catalog_manager.cc:4841] TS 91bc72a32d1c4a729cfee1ebbaeba917 (127.4.217.3:34263): tablet 6d5ea00775a845a2a67db9de58a9f305 (table test-table [id=72c84b48d6ac41e5aac3c1a94566eb29]) successfully deleted
I20260808 04:45:34.680912 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 16454
I20260808 04:45:34.681146 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:34.680844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:36669 (local address 127.0.0.1:44330)
0808 04:45:34.680980 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:34.681041 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:36669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:45:34.681344 15827 catalog_manager.cc:4584] TS 54e274a0492549cb8762a91403cff09d (127.4.217.1:36669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d5ea00775a845a2a67db9de58a9f305: Network error: Client connection negotiation failed: client connection to 127.4.217.1:36669: connect: Connection refused (error 111)
I20260808 04:45:34.698061 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 16688
I20260808 04:45:34.713030 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 15615
I20260808 04:45:34.730419 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 15803
2026-08-08T04:45:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5623 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:34 dist-test-slave-2kcd krb5kdc[17093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:34 dist-test-slave-2kcd krb5kdc[17093](info): set up 2 sockets
Aug 08 04:45:34 dist-test-slave-2kcd krb5kdc[17093](info): commencing operation
krb5kdc: starting...
W20260808 04:45:36.001082 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:36 dist-test-slave-2kcd krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.62.keytab.
Entry for principal oryx/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.62.keytab.
I20260808 04:45:36.122162 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:44571
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.62.keytab
--principal=oryx/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:44571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:36.235901 17116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:36.236198 17116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:36.236267 17116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:36.240161 17116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:36.240247 17116 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:45:36.240278 17116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:36.240324 17116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:36.240360 17116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:36.240394 17116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:36.245535 17116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:44571
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.62.keytab
--principal=oryx/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:44571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:36.246546 17116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:36.247558 17116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:36.258575 17130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:36.258628 17128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:36.258847 17116 server_base.cc:1034] running on GCE node
W20260808 04:45:36.258571 17125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:36.259755 17116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:36.260882 17116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:36.262070 17116 hybrid_clock.cc:648] HybridClock initialized: now 1786164336262057 us; error 31 us; skew 500 ppm
Aug 08 04:45:36 dist-test-slave-2kcd krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:36.264734 17116 init.cc:377] Logged in from keytab as oryx/127.4.217.62@KRBTEST.COM (short username oryx)
I20260808 04:45:36.265652 17116 webserver.cc:460] Webserver started at http://127.4.217.62:39519/ using document root <none> and password file <none>
I20260808 04:45:36.266220 17116 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:36.266310 17116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:36.266538 17116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:36.268597 17116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "f289c6be5b3b4358b364f70cf8cc221e"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "dcdd35626a49d62bb250b2dee1a7a78c"
server_key_iv: "78bc47bf79569f4d8757952540900173"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.269217 17116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "f289c6be5b3b4358b364f70cf8cc221e"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "dcdd35626a49d62bb250b2dee1a7a78c"
server_key_iv: "78bc47bf79569f4d8757952540900173"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.272826 17116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:45:36.275369 17138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:36.276515 17116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:45:36.276654 17116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "f289c6be5b3b4358b364f70cf8cc221e"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "dcdd35626a49d62bb250b2dee1a7a78c"
server_key_iv: "78bc47bf79569f4d8757952540900173"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.276762 17116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:36.303139 17116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:36.305974 17116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:36.306196 17116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:36.314248 17116 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:44571
I20260808 04:45:36.314280 17221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:44571 every 8 connection(s)
I20260808 04:45:36.315201 17116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:36.318048 17226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:36.318583 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 17116
I20260808 04:45:36.318738 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:36.319087 4964 external_mini_cluster.cc:1428] Setting key f6f71f484063fc01987a98f4cb8d8da6
I20260808 04:45:36.324848 17226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e: Bootstrap starting.
I20260808 04:45:36.327270 17226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e: Neither blocks nor log segments found. Creating new log.
Aug 08 04:45:36 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:36.328207 17226 log.cc:824] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:36.330413 17226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e: No bootstrap required, opened a new log
I20260808 04:45:36.333853 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:36.320552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.4.217.62:44571)
0808 04:45:36.321022 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:36.321038 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:36.321075 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:45:36.321728 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:36.321747 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:36.322073 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:36.322423 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:36.323214 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.324209 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.325147 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.325619 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.325789 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:36.328571 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:36.328576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:36.328591 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:36.328636 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:36.330993 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.331485 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.331490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.331494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.331611 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.331902 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.331904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.331906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.332282 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:36.332433 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:36.332739 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:45:36.333107 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":147,"threads_started":1}
I20260808 04:45:36.334007 17226 raft_consensus.cc:348] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f289c6be5b3b4358b364f70cf8cc221e" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 44571 } }
I20260808 04:45:36.334287 17226 raft_consensus.cc:374] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:36.334363 17226 raft_consensus.cc:729] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f289c6be5b3b4358b364f70cf8cc221e, State: Initialized, Role: FOLLOWER
I20260808 04:45:36.334790 17226 consensus_queue.cc:260] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f289c6be5b3b4358b364f70cf8cc221e" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 44571 } }
I20260808 04:45:36.334951 17226 raft_consensus.cc:388] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:36.335022 17226 raft_consensus.cc:482] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:36.335117 17226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:36.336244 17226 raft_consensus.cc:504] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f289c6be5b3b4358b364f70cf8cc221e" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 44571 } }
I20260808 04:45:36.336586 17226 leader_election.cc:302] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f289c6be5b3b4358b364f70cf8cc221e; no voters:
I20260808 04:45:36.336937 17226 leader_election.cc:288] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:36.337038 17236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:36.337286 17236 raft_consensus.cc:686] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [term 1 LEADER]: Becoming Leader. State: Replica: f289c6be5b3b4358b364f70cf8cc221e, State: Running, Role: LEADER
I20260808 04:45:36.337587 17236 consensus_queue.cc:237] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f289c6be5b3b4358b364f70cf8cc221e" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 44571 } }
I20260808 04:45:36.338199 17226 sys_catalog.cc:564] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:36.339311 17238 sys_catalog.cc:455] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f289c6be5b3b4358b364f70cf8cc221e. Latest consensus state: current_term: 1 leader_uuid: "f289c6be5b3b4358b364f70cf8cc221e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f289c6be5b3b4358b364f70cf8cc221e" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 44571 } } }
I20260808 04:45:36.339443 17238 sys_catalog.cc:458] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:36.339742 17237 sys_catalog.cc:455] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f289c6be5b3b4358b364f70cf8cc221e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f289c6be5b3b4358b364f70cf8cc221e" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 44571 } } }
I20260808 04:45:36.339840 17237 sys_catalog.cc:458] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:36.340345 17252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:36.343317 17252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:36.349473 17252 catalog_manager.cc:1347] Generated new cluster ID: b1443275fbd943e19b0bdd4ba3b16bee
I20260808 04:45:36.349606 17252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:36.357571 17252 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:36.358642 17252 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:36.363600 17252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e: Generated new TSK 0
I20260808 04:45:36.364374 17252 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab.
Entry for principal oryx/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab.
I20260808 04:45:36.395596 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:44571
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab
--principal=oryx/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:36.506487 17269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:36.506767 17269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:36.506861 17269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:36.510807 17269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:36.510890 17269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:36.510960 17269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:45:36.511036 17269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:36.516044 17269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab
--principal=oryx/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.217.62:44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:36.517156 17269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:36.518165 17269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:36.527763 17278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:36.527755 17279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:36.528002 17269 server_base.cc:1034] running on GCE node
W20260808 04:45:36.527755 17281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:36.528640 17269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:36.529323 17269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:36.530493 17269 hybrid_clock.cc:648] HybridClock initialized: now 1786164336530477 us; error 27 us; skew 500 ppm
Aug 08 04:45:36 dist-test-slave-2kcd krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:36.533339 17269 init.cc:377] Logged in from keytab as oryx/127.4.217.1@KRBTEST.COM (short username oryx)
I20260808 04:45:36.534252 17269 webserver.cc:460] Webserver started at http://127.4.217.1:46865/ using document root <none> and password file <none>
I20260808 04:45:36.534878 17269 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:36.534960 17269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:36.535202 17269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:36.537336 17269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "8702558b9442be93680cd3fd42f2c669"
server_key_iv: "93a635b13e7a176ace710bb872a95003"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.537916 17269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "8702558b9442be93680cd3fd42f2c669"
server_key_iv: "93a635b13e7a176ace710bb872a95003"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.541967 17269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 04:45:36.544361 17291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:36.545643 17269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:36.545781 17269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "8702558b9442be93680cd3fd42f2c669"
server_key_iv: "93a635b13e7a176ace710bb872a95003"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.545881 17269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:36.568768 17269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:36.572012 17269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:36.572218 17269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:36.573870 17269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:36.573925 17269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:36.573992 17269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:36.574030 17269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:45:36 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:36.585949 17269 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:35037
I20260808 04:45:36.586058 17472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:35037 every 8 connection(s)
I20260808 04:45:36.586908 17269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:36.588495 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:36.574729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:37211 (local address 127.4.217.62:44571)
0808 04:45:36.574871 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:36.574875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:36.575779 (+ 904us) server_negotiation.cc:407] Connection header received
0808 04:45:36.577232 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:36.577236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:36.577294 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:36.577405 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:36.578737 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.579685 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.581207 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.581661 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.581790 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:36.584607 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:36.584612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:36.584615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:36.584649 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:36.586290 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.586793 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.586799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.586801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.586890 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.587249 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.587254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.587256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.587501 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:36.587600 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:36.588234 (+ 634us) server_negotiation.cc:299] Negotiation successful
0808 04:45:36.588349 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:45:36.589285 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:36.575068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:44571 (local address 127.4.217.1:37211)
0808 04:45:36.575590 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:36.575652 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:36.576957 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:36.577721 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:36.577727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:36.578081 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:36.578573 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:36.578587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.579930 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.579933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:36.581097 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:36.581107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.581881 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.581885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:36.582078 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:36.582734 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:36.582753 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:36.584437 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:36.586405 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:36.586413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:36.586428 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:36.586688 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:36.586982 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:36.586986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:36.586988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:36.587153 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:36.587651 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:36.587657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:36.587959 (+ 302us) client_negotiation.cc:769] Sending connection context
0808 04:45:36.588162 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 04:45:36.588416 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":120,"threads_started":1}
I20260808 04:45:36.590610 17473 heartbeater.cc:347] Connected to a master server at 127.4.217.62:44571
I20260808 04:45:36.590760 17473 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:36.591318 17473 heartbeater.cc:511] Master 127.4.217.62:44571 requested a full tablet report, sending...
I20260808 04:45:36.592818 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 17269
I20260808 04:45:36.592960 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:36.593174 17160 ts_manager.cc:194] Registered new tserver with Master: 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037)
I20260808 04:45:36.593334 4964 external_mini_cluster.cc:1428] Setting key ad287fa1be6894b94226f9d768d8ec43
I20260808 04:45:36.594564 17160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:37211
I20260808 04:45:36.600613 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:36.593763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:46337 (local address 127.4.217.62:44571)
0808 04:45:36.593895 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:36.593899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:36.593913 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:45:36.593947 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:36.593950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:36.593990 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:36.594083 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:36.594594 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.595290 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.596159 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.596368 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.596473 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:36.597614 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:36.597620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:36.597622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:36.597647 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:36.598995 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.599422 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.599425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.599427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.599506 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.599805 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.599809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.599811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.599980 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:36.600095 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:36.600349 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 04:45:36.600455 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":55}
I20260808 04:45:36.600648 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:36.592992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:44571 (local address 127.4.217.1:46337)
0808 04:45:36.593352 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:36.593368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:36.593465 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:36.594122 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:36.594125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:36.594321 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:36.594469 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:36.594476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.595427 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.595430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:36.596028 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:36.596038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.596481 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.596484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:36.596587 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:36.597112 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:36.597124 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:36.597443 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:36.599117 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:36.599121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:36.599124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:36.599323 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:36.599608 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:36.599611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:36.599613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:36.599707 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:36.600085 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:36.600088 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:36.600187 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 04:45:36.600293 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:45:36.600441 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":23,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab.
Entry for principal oryx/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab.
I20260808 04:45:36.621896 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:44571
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab
--principal=oryx/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:36.735592 17489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:36.735836 17489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:36.735913 17489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:36.740059 17489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:36.740113 17489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:36.740176 17489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:45:36.740250 17489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:36.745522 17489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab
--principal=oryx/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.217.62:44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:36.746521 17489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:36.747524 17489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:36.757238 17505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:36.757333 17496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:36.757294 17508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:36.757515 17489 server_base.cc:1034] running on GCE node
I20260808 04:45:36.757849 17489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:36.758455 17489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:36.759626 17489 hybrid_clock.cc:648] HybridClock initialized: now 1786164336759612 us; error 26 us; skew 500 ppm
Aug 08 04:45:36 dist-test-slave-2kcd krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:36.762490 17489 init.cc:377] Logged in from keytab as oryx/127.4.217.2@KRBTEST.COM (short username oryx)
I20260808 04:45:36.763403 17489 webserver.cc:460] Webserver started at http://127.4.217.2:33813/ using document root <none> and password file <none>
I20260808 04:45:36.763967 17489 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:36.764026 17489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:36.764278 17489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:36.766490 17489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "98a6325ceaec433288922dd72fa4b3db"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "7813fde8bcfe8f8a0b6367d5d2b73fcc"
server_key_iv: "c8db1cfca64e03bc36955731a5012488"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.767107 17489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "98a6325ceaec433288922dd72fa4b3db"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "7813fde8bcfe8f8a0b6367d5d2b73fcc"
server_key_iv: "c8db1cfca64e03bc36955731a5012488"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.771157 17489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 04:45:36.773669 17519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:36.775065 17489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:45:36.775197 17489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "98a6325ceaec433288922dd72fa4b3db"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "7813fde8bcfe8f8a0b6367d5d2b73fcc"
server_key_iv: "c8db1cfca64e03bc36955731a5012488"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.775312 17489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:36.790709 17489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:36.793664 17489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:36.793870 17489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:36.795361 17489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:36.795431 17489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:36.795506 17489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:36.795549 17489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:45:36 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:36.806566 17489 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:41421
I20260808 04:45:36.806646 17696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:41421 every 8 connection(s)
I20260808 04:45:36.807691 17489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:36.808701 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 17489
I20260808 04:45:36.808879 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:36.809167 4964 external_mini_cluster.cc:1428] Setting key 5239d7c296d4a5a021494dfff89d15e6
I20260808 04:45:36.811144 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:36.796211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:39123 (local address 127.4.217.62:44571)
0808 04:45:36.796347 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:36.796350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:36.797447 (+ 1097us) server_negotiation.cc:407] Connection header received
0808 04:45:36.798600 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:36.798604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:36.798672 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:36.798764 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:36.800435 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.801184 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.802325 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.802559 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.802673 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:36.806684 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:36.806688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:36.806692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:36.806720 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:36.808372 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.809041 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.809046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.809047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.809146 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.809611 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.809614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.809615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.809872 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:36.809983 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:36.810916 (+ 933us) server_negotiation.cc:299] Negotiation successful
0808 04:45:36.811027 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:45:36.812222 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:36.796750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:44571 (local address 127.4.217.2:39123)
0808 04:45:36.797281 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:36.797331 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:36.798357 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:36.799012 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:36.799024 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:36.799544 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:36.800253 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:36.800276 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.801322 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.801326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:36.802191 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:36.802201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.802727 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.802732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:36.802930 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:36.803522 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:36.803542 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:36.806487 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:36.808515 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:36.808520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:36.808543 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:36.808921 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:36.809291 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:36.809296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:36.809298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:36.809483 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:36.810395 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:36.810400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:36.810649 (+ 249us) client_negotiation.cc:769] Sending connection context
0808 04:45:36.810847 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 04:45:36.811196 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":143,"threads_started":1}
I20260808 04:45:36.813611 17697 heartbeater.cc:347] Connected to a master server at 127.4.217.62:44571
I20260808 04:45:36.813769 17697 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:36.814368 17697 heartbeater.cc:511] Master 127.4.217.62:44571 requested a full tablet report, sending...
I20260808 04:45:36.815397 17159 ts_manager.cc:194] Registered new tserver with Master: 98a6325ceaec433288922dd72fa4b3db (127.4.217.2:41421)
I20260808 04:45:36.816115 17159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:39123
WARNING: no policy specified for oryx/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:45:36.823371 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:36.815799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:47453 (local address 127.4.217.62:44571)
0808 04:45:36.815900 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:36.815904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:36.816204 (+ 300us) server_negotiation.cc:407] Connection header received
0808 04:45:36.816325 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:36.816330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:36.816380 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:36.816469 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:36.817016 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.817497 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.818781 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.818983 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.819065 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:36.820344 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:36.820347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:36.820349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:36.820370 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:36.821755 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.822224 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.822227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.822228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.822339 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:36.822640 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:36.822642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:36.822643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:36.822840 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:36.822920 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:36.823145 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 04:45:36.823235 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:45:36.823446 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:36.815733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:44571 (local address 127.4.217.2:47453)
0808 04:45:36.816106 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:36.816123 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:36.816227 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:36.816512 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:36.816518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:36.816753 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:36.816901 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:36.816910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.817641 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.817644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:36.818652 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:36.818668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:36.819102 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:36.819106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:36.819238 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:36.819729 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:36.819751 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:36.820202 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:36.821868 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:36.821871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:36.821874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:36.822129 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:36.822426 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:36.822431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:36.822435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:36.822542 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:36.822926 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:36.822929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:36.823027 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 04:45:36.823111 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:45:36.823216 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab.
Entry for principal oryx/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab.
I20260808 04:45:36.841809 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:44571
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab
--principal=oryx/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:36.959323 17714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:36.959563 17714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:36.959606 17714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:36.963452 17714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:36.963500 17714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:36.963536 17714 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:45:36.963567 17714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:36.968978 17714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab
--principal=oryx/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.217.62:44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:36.969916 17714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:36.970858 17714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:36.980309 17723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:36.980319 17727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:36.980408 17724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:36.980690 17714 server_base.cc:1034] running on GCE node
I20260808 04:45:36.981132 17714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:36.981685 17714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:36.982846 17714 hybrid_clock.cc:648] HybridClock initialized: now 1786164336982833 us; error 27 us; skew 500 ppm
Aug 08 04:45:36 dist-test-slave-2kcd krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:36.985401 17714 init.cc:377] Logged in from keytab as oryx/127.4.217.3@KRBTEST.COM (short username oryx)
I20260808 04:45:36.986214 17714 webserver.cc:460] Webserver started at http://127.4.217.3:45519/ using document root <none> and password file <none>
I20260808 04:45:36.986743 17714 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:36.986804 17714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:36.987028 17714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:36.989085 17714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "46119d7c36344798813f510a02cb05b5"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "7aa59da8cfad6939aeb1a3a4b2148e31"
server_key_iv: "c1f63c6877b2aab6a3f355ba89950121"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.989639 17714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "46119d7c36344798813f510a02cb05b5"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "7aa59da8cfad6939aeb1a3a4b2148e31"
server_key_iv: "c1f63c6877b2aab6a3f355ba89950121"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.993427 17714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:45:36.995725 17738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:36.996923 17714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:36.997053 17714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "46119d7c36344798813f510a02cb05b5"
format_stamp: "Formatted at 2026-08-08 04:45:36 on dist-test-slave-2kcd"
server_key: "7aa59da8cfad6939aeb1a3a4b2148e31"
server_key_iv: "c1f63c6877b2aab6a3f355ba89950121"
server_key_version: "encryptionkey@0"
I20260808 04:45:36.997150 17714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:37.010336 17714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:37.013121 17714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:37.013316 17714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:37.014911 17714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:37.014981 17714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:37.015057 17714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:37.015105 17714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:37.026250 17714 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:46163
I20260808 04:45:37.026325 17935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:46163 every 8 connection(s)
I20260808 04:45:37.027406 17714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:37.028921 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 17714
I20260808 04:45:37.029069 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:37.029381 4964 external_mini_cluster.cc:1428] Setting key 508fb782e5874313849b898e983ea41b
I20260808 04:45:37.029822 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.015689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:38169 (local address 127.4.217.62:44571)
0808 04:45:37.015869 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.015876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.016675 (+ 799us) server_negotiation.cc:407] Connection header received
0808 04:45:37.017671 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.017674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.017736 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.017839 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:37.019599 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.020461 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.022368 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.022676 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.022800 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.025688 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:37.025692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:37.025694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:37.025719 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:37.027291 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.027867 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.027874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.027875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.027970 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.028367 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.028370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.028372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.028674 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:37.028812 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.029493 (+ 681us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.029644 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:45:37.030690 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.016039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:44571 (local address 127.4.217.3:38169)
0808 04:45:37.016487 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.016539 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.017453 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.018179 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.018189 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.018730 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:37.019417 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.019442 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.020625 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.020629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.022222 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.022238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.022856 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.022861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.023071 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.023756 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:37.023772 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:37.025508 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:37.027415 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:37.027421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:37.027436 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:37.027758 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:37.028081 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:37.028085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:37.028087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:37.028258 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:37.028865 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:37.028872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:37.029223 (+ 351us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.029381 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.029686 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":131,"threads_started":1}
I20260808 04:45:37.031944 17936 heartbeater.cc:347] Connected to a master server at 127.4.217.62:44571
I20260808 04:45:37.032111 17936 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:37.032616 17936 heartbeater.cc:511] Master 127.4.217.62:44571 requested a full tablet report, sending...
I20260808 04:45:37.033772 17159 ts_manager.cc:194] Registered new tserver with Master: 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163)
I20260808 04:45:37.034511 17159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:38169
I20260808 04:45:37.040997 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.034192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:44571 (local address 127.4.217.3:59347)
0808 04:45:37.034486 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.034499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.034584 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.034845 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.034848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.035045 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:37.035185 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.035192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.035913 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.035917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.036685 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.036695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.037149 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.037152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.037220 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.037743 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:37.037759 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:37.038074 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:37.039689 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:37.039691 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:37.039693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:37.039876 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:37.040160 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:37.040162 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:37.040164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:37.040241 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:37.040555 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:37.040557 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:37.040629 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.040709 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.040872 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":127,"threads_started":1}
I20260808 04:45:37.041026 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.034263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:59347 (local address 127.4.217.62:44571)
0808 04:45:37.034372 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.034376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.034571 (+ 195us) server_negotiation.cc:407] Connection header received
0808 04:45:37.034678 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.034681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.034722 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.034809 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:37.035306 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.035780 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.036828 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.037052 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.037131 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.038186 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:37.038188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:37.038190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:37.038210 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:37.039590 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.039961 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.039964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.039965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.040088 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.040318 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.040320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.040322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.040471 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:37.040550 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.040764 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.040906 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:45:37.043315 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:45:37.068428 17234 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:37.062032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.4.217.62:44571)
0808 04:45:37.062238 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.062244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.062337 (+ 93us) server_negotiation.cc:407] Connection header received
0808 04:45:37.062544 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.062547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.062591 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.062679 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:37.063393 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.063926 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.064814 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.065038 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.065145 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.068280 (+ 3135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45960: BlockingRecv error: recv got EOF from 127.0.0.1:45960 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:37.077674 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.070076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.4.217.62:44571)
0808 04:45:37.070204 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.070208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.070318 (+ 110us) server_negotiation.cc:407] Connection header received
0808 04:45:37.070409 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.070411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.070448 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.070524 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:37.071025 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.071486 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.072210 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.072403 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.072475 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.074688 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:37.074691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:37.074693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:37.074715 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:37.076124 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.076523 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.076526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.076527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.076610 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.076973 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.076976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.076977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.077159 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:37.077269 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.077462 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.077557 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:45:37.080667 17159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:37.083858 17159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:37.098493 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.093076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:46163 (local address 127.0.0.1:41124)
0808 04:45:37.093816 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.093827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.093912 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.094385 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.094388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.094406 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:37.094531 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.094538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.095642 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.095645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.097019 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.097031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.097932 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.097936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.098054 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.098099 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.098231 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.098351 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":19,"thread_start_us":93,"threads_started":1}
I20260808 04:45:37.098510 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.093683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.4.217.3:46163)
0808 04:45:37.094036 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.094047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.094065 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:45:37.094127 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.094130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.094259 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.094405 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:37.094649 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.095505 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.097161 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.097797 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.097918 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.098100 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.098262 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.098352 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":122,"threads_started":1}
I20260808 04:45:37.098846 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.092526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.4.217.2:41421)
0808 04:45:37.093191 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.093204 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.093230 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:45:37.093312 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.093318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.093535 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.094104 (+ 569us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:37.094248 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.095289 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.097131 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.097852 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.097959 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.098117 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.098574 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.098694 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":125,"threads_started":1}
I20260808 04:45:37.098887 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.092329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:41421 (local address 127.0.0.1:49110)
0808 04:45:37.092691 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.092746 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.092902 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.093726 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.093732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.093759 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:37.093916 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.093923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.095407 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.095411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.097007 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.097017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.098188 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.098192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.098295 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.098322 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.098389 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.098459 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":307,"thread_start_us":90,"threads_started":1}
I20260808 04:45:37.099116 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.092717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:35037 (local address 127.0.0.1:47900)
0808 04:45:37.093512 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.093523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.093634 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.094682 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.094684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.094709 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:37.094811 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.094817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.096079 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.096083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.097311 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.097319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.098525 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.098528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.098770 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.098793 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.098870 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.098956 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":56,"thread_start_us":162,"threads_started":1}
I20260808 04:45:37.099567 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.093197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.4.217.1:35037)
0808 04:45:37.094130 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.094136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.094158 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:45:37.094236 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.094241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.094424 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.094580 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:37.094931 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.095941 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.097435 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.098226 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.098323 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.098460 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.099342 (+ 882us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.099422 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":95,"threads_started":1}
I20260808 04:45:37.102240 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 5386eebd736342c88f311d06764a568f (DEFAULT_TABLE table=test-table [id=97400665238a42db8673122b1f53744c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.102272 17822 tablet_service.cc:1467] Processing CreateTablet for tablet 5386eebd736342c88f311d06764a568f (DEFAULT_TABLE table=test-table [id=97400665238a42db8673122b1f53744c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.102259 17375 tablet_service.cc:1467] Processing CreateTablet for tablet 5386eebd736342c88f311d06764a568f (DEFAULT_TABLE table=test-table [id=97400665238a42db8673122b1f53744c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.103263 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5386eebd736342c88f311d06764a568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.103313 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5386eebd736342c88f311d06764a568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.103313 17822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5386eebd736342c88f311d06764a568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.109897 17979 tablet_bootstrap.cc:492] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: Bootstrap starting.
I20260808 04:45:37.111223 17982 tablet_bootstrap.cc:492] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: Bootstrap starting.
I20260808 04:45:37.111773 17980 tablet_bootstrap.cc:492] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Bootstrap starting.
I20260808 04:45:37.112030 17979 tablet_bootstrap.cc:654] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.112778 17979 log.cc:824] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:37.113468 17982 tablet_bootstrap.cc:654] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.113893 17980 tablet_bootstrap.cc:654] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.114477 17982 log.cc:824] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:37.114773 17980 log.cc:824] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:37.114884 17979 tablet_bootstrap.cc:492] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: No bootstrap required, opened a new log
I20260808 04:45:37.115093 17979 ts_tablet_manager.cc:1397] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:45:37.116254 17982 tablet_bootstrap.cc:492] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: No bootstrap required, opened a new log
I20260808 04:45:37.116454 17982 ts_tablet_manager.cc:1397] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:45:37.116674 17980 tablet_bootstrap.cc:492] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: No bootstrap required, opened a new log
I20260808 04:45:37.116888 17980 ts_tablet_manager.cc:1397] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:45:37.119110 17979 raft_consensus.cc:348] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } }
I20260808 04:45:37.119465 17979 raft_consensus.cc:374] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.119326 17982 raft_consensus.cc:348] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } }
I20260808 04:45:37.119568 17982 raft_consensus.cc:374] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.119611 17979 raft_consensus.cc:729] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a6325ceaec433288922dd72fa4b3db, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.119695 17982 raft_consensus.cc:729] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46119d7c36344798813f510a02cb05b5, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.120121 17979 consensus_queue.cc:260] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } }
I20260808 04:45:37.120196 17982 consensus_queue.cc:260] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } }
I20260808 04:45:37.120841 17697 heartbeater.cc:503] Master 127.4.217.62:44571 was elected leader, sending a full tablet report...
I20260808 04:45:37.121165 17979 ts_tablet_manager.cc:1428] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:45:37.121060 17980 raft_consensus.cc:348] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } }
I20260808 04:45:37.121263 17982 ts_tablet_manager.cc:1428] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 04:45:37.121335 17980 raft_consensus.cc:374] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.121403 17980 raft_consensus.cc:729] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e61140b3a2f4e4fb8eda46f59eb6ab8, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.121630 17936 heartbeater.cc:503] Master 127.4.217.62:44571 was elected leader, sending a full tablet report...
I20260808 04:45:37.121819 17980 consensus_queue.cc:260] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } }
I20260808 04:45:37.122416 17473 heartbeater.cc:503] Master 127.4.217.62:44571 was elected leader, sending a full tablet report...
I20260808 04:45:37.122666 17980 ts_tablet_manager.cc:1428] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:45:37.255916 17991 raft_consensus.cc:482] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:37.256148 17991 raft_consensus.cc:504] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } }
I20260808 04:45:37.257350 17991 leader_election.cc:288] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98a6325ceaec433288922dd72fa4b3db (127.4.217.2:41421), 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037)
I20260808 04:45:37.261775 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.257728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:60865 (local address 127.4.217.2:41421)
0808 04:45:37.257862 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.257867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.257965 (+ 98us) server_negotiation.cc:407] Connection header received
0808 04:45:37.258100 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.258104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.258177 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.258292 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:37.258591 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.259398 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.260621 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.261215 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.261296 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.261351 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.261559 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.261618 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:45:37.261796 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.257625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:41421 (local address 127.4.217.3:60865)
0808 04:45:37.257863 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.257879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.257991 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.258299 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.258301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.258335 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:37.258464 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.258474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.259508 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.259511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.260486 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.260496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.261322 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.261325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.261429 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.261456 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.261536 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.261607 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":1}
I20260808 04:45:37.262271 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.257728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:39503 (local address 127.4.217.1:35037)
0808 04:45:37.257862 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.257867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.258232 (+ 365us) server_negotiation.cc:407] Connection header received
0808 04:45:37.258381 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.258384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.258440 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.258527 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:37.259264 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.259936 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.261104 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.261662 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.261763 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.261835 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.262078 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.262158 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:45:37.262530 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5386eebd736342c88f311d06764a568f" candidate_uuid: "46119d7c36344798813f510a02cb05b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a6325ceaec433288922dd72fa4b3db" is_pre_election: true
I20260808 04:45:37.262624 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.257626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:35037 (local address 127.4.217.3:39503)
0808 04:45:37.258118 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.258133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.258251 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.258568 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.258571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.258590 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:37.258721 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.258728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.260057 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.260060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.260978 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.260989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.261788 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.261795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.261901 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.261933 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.262433 (+ 500us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.262507 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":43,"thread_start_us":123,"threads_started":1}
I20260808 04:45:37.262868 17625 raft_consensus.cc:2454] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46119d7c36344798813f510a02cb05b5 in term 0.
I20260808 04:45:37.263314 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5386eebd736342c88f311d06764a568f" candidate_uuid: "46119d7c36344798813f510a02cb05b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" is_pre_election: true
I20260808 04:45:37.263420 17741 leader_election.cc:302] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46119d7c36344798813f510a02cb05b5, 98a6325ceaec433288922dd72fa4b3db; no voters:
I20260808 04:45:37.263610 17406 raft_consensus.cc:2454] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46119d7c36344798813f510a02cb05b5 in term 0.
I20260808 04:45:37.263752 17991 raft_consensus.cc:2781] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:37.263870 17991 raft_consensus.cc:482] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:37.263943 17991 raft_consensus.cc:3037] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.265076 17991 raft_consensus.cc:504] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } }
I20260808 04:45:37.265367 17991 leader_election.cc:288] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [CANDIDATE]: Term 1 election: Requested vote from peers 98a6325ceaec433288922dd72fa4b3db (127.4.217.2:41421), 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037)
I20260808 04:45:37.265717 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5386eebd736342c88f311d06764a568f" candidate_uuid: "46119d7c36344798813f510a02cb05b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8"
I20260808 04:45:37.265724 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5386eebd736342c88f311d06764a568f" candidate_uuid: "46119d7c36344798813f510a02cb05b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a6325ceaec433288922dd72fa4b3db"
I20260808 04:45:37.265823 17406 raft_consensus.cc:3037] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.265854 17625 raft_consensus.cc:3037] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.266875 17406 raft_consensus.cc:2454] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46119d7c36344798813f510a02cb05b5 in term 1.
I20260808 04:45:37.267223 17739 leader_election.cc:302] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46119d7c36344798813f510a02cb05b5, 7e61140b3a2f4e4fb8eda46f59eb6ab8; no voters:
I20260808 04:45:37.267442 17625 raft_consensus.cc:2454] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46119d7c36344798813f510a02cb05b5 in term 1.
I20260808 04:45:37.267658 17991 raft_consensus.cc:2781] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:37.267879 17991 raft_consensus.cc:686] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 1 LEADER]: Becoming Leader. State: Replica: 46119d7c36344798813f510a02cb05b5, State: Running, Role: LEADER
I20260808 04:45:37.268217 17991 consensus_queue.cc:237] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } }
I20260808 04:45:37.271502 17159 catalog_manager.cc:5495] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46119d7c36344798813f510a02cb05b5 (127.4.217.3). New cstate: current_term: 1 leader_uuid: "46119d7c36344798813f510a02cb05b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } health_report { overall_health: UNKNOWN } } }
W20260808 04:45:37.279078 17937 tablet.cc:2369] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:37.290283 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.286881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.4.217.3:46163)
0808 04:45:37.287018 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.287022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.287037 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:45:37.287188 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.287191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.287245 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.287337 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:37.287535 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.288026 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.288897 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.289194 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.289278 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.289515 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:37.289930 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:37.290025 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.290094 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.290180 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:45:37.295928 17625 raft_consensus.cc:1261] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Refusing update from remote peer 46119d7c36344798813f510a02cb05b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:37.295940 17406 raft_consensus.cc:1261] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Refusing update from remote peer 46119d7c36344798813f510a02cb05b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:37.296631 17991 consensus_queue.cc:1035] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:37.296890 17995 consensus_queue.cc:1035] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:37.308074 18005 mvcc.cc:205] Tried to move back new op lower bound from 7316129125557370880 to 7316129125453840384. Current Snapshot: MvccSnapshot[applied={T|T < 7316129125557370880 or (T in {7316129125557370880})}]
I20260808 04:45:37.313530 17159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:37.313688 17159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:37.316212 17159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e: Sending DeleteTablet for 3 replicas of tablet 5386eebd736342c88f311d06764a568f
I20260808 04:45:37.317062 17600 tablet_service.cc:1514] Processing DeleteTablet for tablet 5386eebd736342c88f311d06764a568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:37 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:49110
I20260808 04:45:37.317065 17375 tablet_service.cc:1514] Processing DeleteTablet for tablet 5386eebd736342c88f311d06764a568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:37 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260808 04:45:37.317070 17822 tablet_service.cc:1514] Processing DeleteTablet for tablet 5386eebd736342c88f311d06764a568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:37 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:41124
I20260808 04:45:37.317600 18019 tablet_replica.cc:330] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: stopping tablet replica
I20260808 04:45:37.317929 18019 raft_consensus.cc:2229] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:37.318403 18019 raft_consensus.cc:2258] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:37.318554 17159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:37.318988 17159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:37.320127 18019 ts_tablet_manager.cc:1905] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:37.323311 18019 ts_tablet_manager.cc:1918] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:37.323490 18019 log.cc:1196] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/5386eebd736342c88f311d06764a568f
I20260808 04:45:37.323928 18019 ts_tablet_manager.cc:1939] T 5386eebd736342c88f311d06764a568f P 46119d7c36344798813f510a02cb05b5: Deleting consensus metadata
I20260808 04:45:37.324095 18018 tablet_replica.cc:330] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: stopping tablet replica
I20260808 04:45:37.324466 18018 raft_consensus.cc:2229] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:37.325686 17143 catalog_manager.cc:4841] TS 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163): tablet 5386eebd736342c88f311d06764a568f (table test-table [id=97400665238a42db8673122b1f53744c]) successfully deleted
I20260808 04:45:37.325970 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 42ddf5d6951d49e8813c9b10879a5b7c (DEFAULT_TABLE table=test-table [id=d2123824f82c4083bf8fa737db0a3391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.326277 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ddf5d6951d49e8813c9b10879a5b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.326221 17822 tablet_service.cc:1467] Processing CreateTablet for tablet 42ddf5d6951d49e8813c9b10879a5b7c (DEFAULT_TABLE table=test-table [id=d2123824f82c4083bf8fa737db0a3391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.326474 17822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ddf5d6951d49e8813c9b10879a5b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.328845 17979 tablet_bootstrap.cc:492] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: Bootstrap starting.
I20260808 04:45:37.328908 17982 tablet_bootstrap.cc:492] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: Bootstrap starting.
I20260808 04:45:37.329855 17982 tablet_bootstrap.cc:654] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.329855 17979 tablet_bootstrap.cc:654] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.325292 17375 tablet_service.cc:1467] Processing CreateTablet for tablet 42ddf5d6951d49e8813c9b10879a5b7c (DEFAULT_TABLE table=test-table [id=d2123824f82c4083bf8fa737db0a3391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.331609 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ddf5d6951d49e8813c9b10879a5b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.331677 18017 tablet_replica.cc:330] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: stopping tablet replica
I20260808 04:45:37.331729 18018 raft_consensus.cc:2258] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:37.331981 17982 tablet_bootstrap.cc:492] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: No bootstrap required, opened a new log
I20260808 04:45:37.331990 18017 raft_consensus.cc:2229] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:37.332082 17982 ts_tablet_manager.cc:1397] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:37.332257 18017 raft_consensus.cc:2258] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:37.332561 17982 raft_consensus.cc:348] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.332670 17982 raft_consensus.cc:374] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.332701 17982 raft_consensus.cc:729] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46119d7c36344798813f510a02cb05b5, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.332840 17982 consensus_queue.cc:260] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.333086 17982 ts_tablet_manager.cc:1428] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:37.333222 17979 tablet_bootstrap.cc:492] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: No bootstrap required, opened a new log
I20260808 04:45:37.333304 17979 ts_tablet_manager.cc:1397] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 04:45:37.333742 17979 raft_consensus.cc:348] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.333847 17979 raft_consensus.cc:374] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.333882 17979 raft_consensus.cc:729] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a6325ceaec433288922dd72fa4b3db, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.334010 17979 consensus_queue.cc:260] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.334438 18018 ts_tablet_manager.cc:1905] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:37.334667 17980 tablet_bootstrap.cc:492] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Bootstrap starting.
I20260808 04:45:37.335587 17979 ts_tablet_manager.cc:1428] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260808 04:45:37.335685 17980 tablet_bootstrap.cc:654] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.336971 17980 tablet_bootstrap.cc:492] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: No bootstrap required, opened a new log
I20260808 04:45:37.337066 17980 ts_tablet_manager.cc:1397] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:37.337743 17980 raft_consensus.cc:348] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.337855 17980 raft_consensus.cc:374] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.337894 17980 raft_consensus.cc:729] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e61140b3a2f4e4fb8eda46f59eb6ab8, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.338155 17980 consensus_queue.cc:260] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.338481 17980 ts_tablet_manager.cc:1428] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:37.336977 18018 ts_tablet_manager.cc:1918] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:37.338837 18018 log.cc:1196] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/5386eebd736342c88f311d06764a568f
I20260808 04:45:37.338953 18017 ts_tablet_manager.cc:1905] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:37.339231 18018 ts_tablet_manager.cc:1939] T 5386eebd736342c88f311d06764a568f P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Deleting consensus metadata
I20260808 04:45:37.340698 17139 catalog_manager.cc:4841] TS 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037): tablet 5386eebd736342c88f311d06764a568f (table test-table [id=97400665238a42db8673122b1f53744c]) successfully deleted
I20260808 04:45:37.341732 18017 ts_tablet_manager.cc:1918] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:37.341849 18017 log.cc:1196] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/5386eebd736342c88f311d06764a568f
I20260808 04:45:37.342139 18017 ts_tablet_manager.cc:1939] T 5386eebd736342c88f311d06764a568f P 98a6325ceaec433288922dd72fa4b3db: Deleting consensus metadata
I20260808 04:45:37.343321 17143 catalog_manager.cc:4841] TS 98a6325ceaec433288922dd72fa4b3db (127.4.217.2:41421): tablet 5386eebd736342c88f311d06764a568f (table test-table [id=97400665238a42db8673122b1f53744c]) successfully deleted
I20260808 04:45:37.496218 18012 raft_consensus.cc:482] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:37.496418 18012 raft_consensus.cc:504] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.496946 18012 leader_election.cc:288] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037), 98a6325ceaec433288922dd72fa4b3db (127.4.217.2:41421)
I20260808 04:45:37.497408 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ddf5d6951d49e8813c9b10879a5b7c" candidate_uuid: "46119d7c36344798813f510a02cb05b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" is_pre_election: true
I20260808 04:45:37.497432 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ddf5d6951d49e8813c9b10879a5b7c" candidate_uuid: "46119d7c36344798813f510a02cb05b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a6325ceaec433288922dd72fa4b3db" is_pre_election: true
I20260808 04:45:37.497583 17406 raft_consensus.cc:2454] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46119d7c36344798813f510a02cb05b5 in term 0.
I20260808 04:45:37.497601 17625 raft_consensus.cc:2454] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46119d7c36344798813f510a02cb05b5 in term 0.
I20260808 04:45:37.497895 17739 leader_election.cc:302] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46119d7c36344798813f510a02cb05b5, 7e61140b3a2f4e4fb8eda46f59eb6ab8; no voters:
I20260808 04:45:37.498078 18012 raft_consensus.cc:2781] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:37.498142 18012 raft_consensus.cc:482] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:37.498168 18012 raft_consensus.cc:3037] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.499174 18012 raft_consensus.cc:504] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.499478 18012 leader_election.cc:288] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [CANDIDATE]: Term 1 election: Requested vote from peers 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037), 98a6325ceaec433288922dd72fa4b3db (127.4.217.2:41421)
I20260808 04:45:37.499789 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ddf5d6951d49e8813c9b10879a5b7c" candidate_uuid: "46119d7c36344798813f510a02cb05b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8"
I20260808 04:45:37.499847 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ddf5d6951d49e8813c9b10879a5b7c" candidate_uuid: "46119d7c36344798813f510a02cb05b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a6325ceaec433288922dd72fa4b3db"
I20260808 04:45:37.499950 17406 raft_consensus.cc:3037] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.499965 17625 raft_consensus.cc:3037] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.500931 17625 raft_consensus.cc:2454] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46119d7c36344798813f510a02cb05b5 in term 1.
I20260808 04:45:37.501144 17406 raft_consensus.cc:2454] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46119d7c36344798813f510a02cb05b5 in term 1.
I20260808 04:45:37.501235 17741 leader_election.cc:302] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46119d7c36344798813f510a02cb05b5, 98a6325ceaec433288922dd72fa4b3db; no voters:
I20260808 04:45:37.501390 18012 raft_consensus.cc:2781] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:37.501497 18012 raft_consensus.cc:686] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 1 LEADER]: Becoming Leader. State: Replica: 46119d7c36344798813f510a02cb05b5, State: Running, Role: LEADER
I20260808 04:45:37.501652 18012 consensus_queue.cc:237] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.503334 17159 catalog_manager.cc:5495] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46119d7c36344798813f510a02cb05b5 (127.4.217.3). New cstate: current_term: 1 leader_uuid: "46119d7c36344798813f510a02cb05b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } health_report { overall_health: HEALTHY } } }
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.62@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:37.551693 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.542943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:44571 (local address 127.0.0.1:45984)
0808 04:45:37.543255 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.543268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.543414 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.543670 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.543673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.544111 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:37.544211 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.544217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.545115 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.545118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.545705 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.545720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.546194 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.546197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.546274 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.546980 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:37.546993 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:37.548490 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:37.550180 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:37.550189 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:37.550201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:37.550424 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:37.550671 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:37.550674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:37.550675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:37.550755 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:37.551167 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:37.551172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:37.551283 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.551377 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.551500 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":123,"threads_started":1}
I20260808 04:45:37.551693 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.542975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.4.217.62:44571)
0808 04:45:37.543125 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.543131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.543348 (+ 217us) server_negotiation.cc:407] Connection header received
0808 04:45:37.543510 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.543513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.543564 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.543649 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:37.544332 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.544970 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.545816 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.546041 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.546132 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.548628 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:37.548632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:37.548634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:37.548656 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:37.550054 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.550512 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.550514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.550515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.550589 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.550880 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.550883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.550885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.551074 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:37.551153 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.551437 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.551555 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":49}
I20260808 04:45:37.555828 17159 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:45984:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:45:37.556566 17159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260808 04:45:37.560178 17699 tablet.cc:2369] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:37.561745 17375 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba5766d5e6e462c95dca58611fb5d4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efd0e73c3d7049fcafb8ef840f9b862a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:45:37.561744 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba5766d5e6e462c95dca58611fb5d4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efd0e73c3d7049fcafb8ef840f9b862a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:45:37.561838 17822 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba5766d5e6e462c95dca58611fb5d4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efd0e73c3d7049fcafb8ef840f9b862a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:45:37.562178 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba5766d5e6e462c95dca58611fb5d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.562179 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba5766d5e6e462c95dca58611fb5d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.562247 17822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba5766d5e6e462c95dca58611fb5d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.564882 17982 tablet_bootstrap.cc:492] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: Bootstrap starting.
I20260808 04:45:37.565485 17979 tablet_bootstrap.cc:492] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: Bootstrap starting.
I20260808 04:45:37.565810 17980 tablet_bootstrap.cc:492] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Bootstrap starting.
I20260808 04:45:37.565919 17982 tablet_bootstrap.cc:654] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.566460 17979 tablet_bootstrap.cc:654] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.566865 17980 tablet_bootstrap.cc:654] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.567309 17982 tablet_bootstrap.cc:492] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: No bootstrap required, opened a new log
I20260808 04:45:37.567399 17982 ts_tablet_manager.cc:1397] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:37.567921 17982 raft_consensus.cc:348] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.568068 17982 raft_consensus.cc:374] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.568123 17980 tablet_bootstrap.cc:492] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: No bootstrap required, opened a new log
I20260808 04:45:37.568120 17979 tablet_bootstrap.cc:492] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: No bootstrap required, opened a new log
I20260808 04:45:37.568126 17982 raft_consensus.cc:729] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46119d7c36344798813f510a02cb05b5, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.568202 17980 ts_tablet_manager.cc:1397] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:37.568202 17979 ts_tablet_manager.cc:1397] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260808 04:45:37.568331 17982 consensus_queue.cc:260] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.568611 18012 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.568656 17979 raft_consensus.cc:348] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.568656 17980 raft_consensus.cc:348] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.568768 17979 raft_consensus.cc:374] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.568768 17980 raft_consensus.cc:374] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.568831 17979 raft_consensus.cc:729] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a6325ceaec433288922dd72fa4b3db, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.568851 17980 raft_consensus.cc:729] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e61140b3a2f4e4fb8eda46f59eb6ab8, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.568969 17980 consensus_queue.cc:260] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.568966 17979 consensus_queue.cc:260] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.569231 17980 ts_tablet_manager.cc:1428] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:45:37.569276 17988 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.569450 17988 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:37.569434 17992 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.569585 17992 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:37.568765 18012 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:37.569774 17982 ts_tablet_manager.cc:1428] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:45:37.570288 17979 ts_tablet_manager.cc:1428] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20260808 04:45:37.588511 17475 tablet.cc:2369] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:45:37.629099 17988 raft_consensus.cc:482] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:37.629245 17988 raft_consensus.cc:504] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.630468 17988 leader_election.cc:288] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037), 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163)
I20260808 04:45:37.634543 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.630714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:46163 (local address 127.4.217.2:40349)
0808 04:45:37.631070 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.631081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.631199 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.631465 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.631467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.631483 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:37.631646 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.631655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.632403 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.632407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.633389 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.633399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.634087 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.634093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.634206 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.634237 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.634307 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.634361 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":46,"thread_start_us":81,"threads_started":1}
I20260808 04:45:37.634565 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.630866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:40349 (local address 127.4.217.3:46163)
0808 04:45:37.631011 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.631018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.631158 (+ 140us) server_negotiation.cc:407] Connection header received
0808 04:45:37.631319 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.631322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.631372 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.631453 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:37.631763 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.632292 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.633517 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.633963 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.634054 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.634143 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.634347 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.634438 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:45:37.634684 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.630801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:53389 (local address 127.4.217.1:35037)
0808 04:45:37.630956 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.630961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.631083 (+ 122us) server_negotiation.cc:407] Connection header received
0808 04:45:37.631246 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.631249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.631298 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.631379 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:45:37.631698 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.632305 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.633629 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.634027 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.634135 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.634204 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.634454 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.634544 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:45:37.634670 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.630644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:35037 (local address 127.4.217.2:53389)
0808 04:45:37.630959 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.630975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.631140 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.631395 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.631398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.631415 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:45:37.631561 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.631568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.632436 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.632439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.633492 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.633506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.634159 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.634165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.634313 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.634339 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.634417 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.634478 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":20,"thread_start_us":93,"threads_started":1}
I20260808 04:45:37.635555 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba5766d5e6e462c95dca58611fb5d4a" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" is_pre_election: true
I20260808 04:45:37.635543 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba5766d5e6e462c95dca58611fb5d4a" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46119d7c36344798813f510a02cb05b5" is_pre_election: true
I20260808 04:45:37.635707 17406 raft_consensus.cc:2454] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 0.
I20260808 04:45:37.635799 17854 raft_consensus.cc:2454] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 0.
I20260808 04:45:37.636077 17520 leader_election.cc:302] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e61140b3a2f4e4fb8eda46f59eb6ab8, 98a6325ceaec433288922dd72fa4b3db; no voters:
I20260808 04:45:37.636360 17988 raft_consensus.cc:2781] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:37.636417 17988 raft_consensus.cc:482] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:37.636444 17988 raft_consensus.cc:3037] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.637485 17988 raft_consensus.cc:504] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.637882 17988 leader_election.cc:288] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 election: Requested vote from peers 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037), 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163)
I20260808 04:45:37.638186 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba5766d5e6e462c95dca58611fb5d4a" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8"
I20260808 04:45:37.638262 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba5766d5e6e462c95dca58611fb5d4a" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46119d7c36344798813f510a02cb05b5"
I20260808 04:45:37.638335 17406 raft_consensus.cc:3037] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.638382 17854 raft_consensus.cc:3037] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.639412 17406 raft_consensus.cc:2454] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 1.
I20260808 04:45:37.639426 17854 raft_consensus.cc:2454] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 1.
I20260808 04:45:37.639698 17520 leader_election.cc:302] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e61140b3a2f4e4fb8eda46f59eb6ab8, 98a6325ceaec433288922dd72fa4b3db; no voters:
I20260808 04:45:37.639884 17988 raft_consensus.cc:2781] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:37.640115 17988 raft_consensus.cc:686] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [term 1 LEADER]: Becoming Leader. State: Replica: 98a6325ceaec433288922dd72fa4b3db, State: Running, Role: LEADER
I20260808 04:45:37.640379 17988 consensus_queue.cc:237] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.641698 18045 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: TxnStatusTablet state changed. Reason: New leader 98a6325ceaec433288922dd72fa4b3db. Latest consensus state: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.641815 18045 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:45:37.642086 18049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:45:37.642243 18049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:45:37.642943 17159 catalog_manager.cc:5495] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 98a6325ceaec433288922dd72fa4b3db (127.4.217.2). New cstate: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:37.671408 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.667466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:41421 (local address 127.0.0.1:49116)
0808 04:45:37.667717 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.667733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.667866 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.668177 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.668180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.668190 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:37.668296 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.668301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.669171 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.669175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.669844 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.669857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.670289 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.670293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.670383 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.670434 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:37.671081 (+ 647us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:37.671099 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.671167 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.671249 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260808 04:45:37.671408 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.667561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.4.217.2:41421)
0808 04:45:37.667702 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.667707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.667797 (+ 90us) server_negotiation.cc:407] Connection header received
0808 04:45:37.668007 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.668010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.668064 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.668164 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:37.668407 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.669063 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.669953 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.670178 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.670287 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.670588 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:37.670958 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:37.671094 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.671178 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.671228 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:45:37.674247 17406 raft_consensus.cc:1261] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Refusing update from remote peer 98a6325ceaec433288922dd72fa4b3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:37.674446 17854 raft_consensus.cc:1261] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Refusing update from remote peer 98a6325ceaec433288922dd72fa4b3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:37.674731 18045 consensus_queue.cc:1035] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:37.675016 17988 consensus_queue.cc:1035] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:37.677062 17992 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: TxnStatusTablet state changed. Reason: New leader 98a6325ceaec433288922dd72fa4b3db. Latest consensus state: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.677196 17992 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:37.676719 18012 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: TxnStatusTablet state changed. Reason: New leader 98a6325ceaec433288922dd72fa4b3db. Latest consensus state: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.677675 18012 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:37.678817 17988 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.678920 17988 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:45:37.679021 18002 mvcc.cc:205] Tried to move back new op lower bound from 7316129127109038080 to 7316129126977683456. Current Snapshot: MvccSnapshot[applied={T|T < 7316129127109038080}]
I20260808 04:45:37.679771 18012 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.679880 18012 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 46119d7c36344798813f510a02cb05b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:37.680253 18045 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.680362 18045 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 98a6325ceaec433288922dd72fa4b3db: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:45:37.681416 17992 tablet_replica.cc:439] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } } }
I20260808 04:45:37.681521 17992 tablet_replica.cc:442] T 5ba5766d5e6e462c95dca58611fb5d4a P 7e61140b3a2f4e4fb8eda46f59eb6ab8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:45:37.693980 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.690134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:41421 (local address 127.4.217.3:32807)
0808 04:45:37.690473 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.690505 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.690601 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.690908 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.690917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.690926 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:37.691060 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.691065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.692085 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.692088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.692720 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.692729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.693348 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.693351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.693419 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.693461 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:37.693747 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:37.693752 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.693801 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.693873 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":101,"threads_started":1}
I20260808 04:45:37.693989 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.690206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:32807 (local address 127.4.217.2:41421)
0808 04:45:37.690322 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.690326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.690574 (+ 248us) server_negotiation.cc:407] Connection header received
0808 04:45:37.690698 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.690702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.690773 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.690863 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:37.691168 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.691954 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.692883 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.693230 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.693335 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.693587 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:37.693663 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:37.693736 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.693813 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.693865 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 04:45:37.700146 17625 raft_consensus.cc:1261] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Refusing update from remote peer 46119d7c36344798813f510a02cb05b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:37.700194 17406 raft_consensus.cc:1261] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Refusing update from remote peer 46119d7c36344798813f510a02cb05b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:37.700562 17991 consensus_queue.cc:1035] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:37.700714 18012 consensus_queue.cc:1035] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:37.704602 18000 mvcc.cc:205] Tried to move back new op lower bound from 7316129127216439296 to 7316129126408101888. Current Snapshot: MvccSnapshot[applied={T|T < 7316129127216439296}]
I20260808 04:45:37.723209 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.719141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:39507 (local address 127.4.217.3:46163)
0808 04:45:37.719264 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.719270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.719475 (+ 205us) server_negotiation.cc:407] Connection header received
0808 04:45:37.719605 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.719607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.719651 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.719715 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:37.720038 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.720852 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.722145 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.722365 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.722453 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.722771 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:37.722838 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:37.722929 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.723029 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.723102 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:45:37.723264 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.719022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:46163 (local address 127.4.217.2:39507)
0808 04:45:37.719386 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:37.719402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:37.719515 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:37.719745 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:37.719748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:37.719756 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:37.719895 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.719901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.721006 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.721009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.721992 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:37.722013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.722481 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.722485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:37.722584 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.722642 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:37.722932 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:37.722939 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:45:37.723014 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:45:37.723097 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":122,"threads_started":1}
I20260808 04:45:37.749313 17159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:37.749518 17159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:37.751835 17159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e: Sending DeleteTablet for 3 replicas of tablet 42ddf5d6951d49e8813c9b10879a5b7c
I20260808 04:45:37.752427 17375 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ddf5d6951d49e8813c9b10879a5b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:37 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260808 04:45:37.752468 17822 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ddf5d6951d49e8813c9b10879a5b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:37 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:41124
I20260808 04:45:37.752545 17600 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ddf5d6951d49e8813c9b10879a5b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:37 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:49110
I20260808 04:45:37.752643 18018 tablet_replica.cc:330] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: stopping tablet replica
I20260808 04:45:37.752650 18019 tablet_replica.cc:330] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: stopping tablet replica
I20260808 04:45:37.752720 18017 tablet_replica.cc:330] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: stopping tablet replica
I20260808 04:45:37.752761 18019 raft_consensus.cc:2229] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:37.752764 18018 raft_consensus.cc:2229] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:37.752841 18017 raft_consensus.cc:2229] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:37.752907 18018 raft_consensus.cc:2258] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:37.752946 18017 raft_consensus.cc:2258] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:37.752975 18019 raft_consensus.cc:2258] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:37.753757 18019 ts_tablet_manager.cc:1905] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:37.753876 18017 ts_tablet_manager.cc:1905] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:37.756222 18017 ts_tablet_manager.cc:1918] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:37.756320 18017 log.cc:1196] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/42ddf5d6951d49e8813c9b10879a5b7c
I20260808 04:45:37.756563 18017 ts_tablet_manager.cc:1939] T 42ddf5d6951d49e8813c9b10879a5b7c P 98a6325ceaec433288922dd72fa4b3db: Deleting consensus metadata
I20260808 04:45:37.756675 18019 ts_tablet_manager.cc:1918] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:37.756768 18019 log.cc:1196] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/42ddf5d6951d49e8813c9b10879a5b7c
I20260808 04:45:37.757081 18019 ts_tablet_manager.cc:1939] T 42ddf5d6951d49e8813c9b10879a5b7c P 46119d7c36344798813f510a02cb05b5: Deleting consensus metadata
I20260808 04:45:37.757308 17143 catalog_manager.cc:4841] TS 98a6325ceaec433288922dd72fa4b3db (127.4.217.2:41421): tablet 42ddf5d6951d49e8813c9b10879a5b7c (table test-table [id=d2123824f82c4083bf8fa737db0a3391]) successfully deleted
I20260808 04:45:37.757736 17143 catalog_manager.cc:4841] TS 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163): tablet 42ddf5d6951d49e8813c9b10879a5b7c (table test-table [id=d2123824f82c4083bf8fa737db0a3391]) successfully deleted
I20260808 04:45:37.758401 18018 ts_tablet_manager.cc:1905] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:37.760565 18018 ts_tablet_manager.cc:1918] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:37.760660 18018 log.cc:1196] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/42ddf5d6951d49e8813c9b10879a5b7c
I20260808 04:45:37.760943 18018 ts_tablet_manager.cc:1939] T 42ddf5d6951d49e8813c9b10879a5b7c P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Deleting consensus metadata
I20260808 04:45:37.761600 17139 catalog_manager.cc:4841] TS 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037): tablet 42ddf5d6951d49e8813c9b10879a5b7c (table test-table [id=d2123824f82c4083bf8fa737db0a3391]) successfully deleted
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:45:37.788726 17234 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:37.782085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.4.217.62:44571)
0808 04:45:37.782240 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.782246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.782430 (+ 184us) server_negotiation.cc:407] Connection header received
0808 04:45:37.782615 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.782623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.782689 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.782799 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:37.783522 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.784053 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.784992 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.785212 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.785318 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.788592 (+ 3274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45996: BlockingRecv error: recv got EOF from 127.0.0.1:45996 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 08 04:45:37 dist-test-slave-2kcd krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM
I20260808 04:45:37.799494 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.790507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.4.217.62:44571)
0808 04:45:37.790625 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.790629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.790864 (+ 235us) server_negotiation.cc:407] Connection header received
0808 04:45:37.790934 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.790936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.790977 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.791049 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:37.791767 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.792315 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.793498 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.793707 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.793786 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.796409 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:37.796413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:37.796415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:37.796441 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:37.797832 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.798274 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.798278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.798280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.798361 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:37.798644 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:37.798647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:37.798649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:37.798828 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:37.798965 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.799203 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.799325 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:45:37.802232 17159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:37.802712 17159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:37.807752 17375 tablet_service.cc:1467] Processing CreateTablet for tablet 266f1be196d8418ba1a105af3fb988c9 (DEFAULT_TABLE table=test-table [id=6aafea149ff541b7a0a13702adba3c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.808012 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 266f1be196d8418ba1a105af3fb988c9 (DEFAULT_TABLE table=test-table [id=6aafea149ff541b7a0a13702adba3c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.808079 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266f1be196d8418ba1a105af3fb988c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.808292 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266f1be196d8418ba1a105af3fb988c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.810647 17980 tablet_bootstrap.cc:492] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Bootstrap starting.
I20260808 04:45:37.811201 17979 tablet_bootstrap.cc:492] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: Bootstrap starting.
I20260808 04:45:37.811612 17980 tablet_bootstrap.cc:654] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.812107 17979 tablet_bootstrap.cc:654] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.813346 17822 tablet_service.cc:1467] Processing CreateTablet for tablet 266f1be196d8418ba1a105af3fb988c9 (DEFAULT_TABLE table=test-table [id=6aafea149ff541b7a0a13702adba3c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.813650 17822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266f1be196d8418ba1a105af3fb988c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.813832 17979 tablet_bootstrap.cc:492] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: No bootstrap required, opened a new log
I20260808 04:45:37.813920 17979 ts_tablet_manager.cc:1397] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 04:45:37.814340 17979 raft_consensus.cc:348] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.814452 17979 raft_consensus.cc:374] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.814492 17979 raft_consensus.cc:729] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a6325ceaec433288922dd72fa4b3db, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.814620 17979 consensus_queue.cc:260] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.814877 17979 ts_tablet_manager.cc:1428] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:45:37.815908 17982 tablet_bootstrap.cc:492] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: Bootstrap starting.
I20260808 04:45:37.816697 17982 tablet_bootstrap.cc:654] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:37.818321 17982 tablet_bootstrap.cc:492] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: No bootstrap required, opened a new log
I20260808 04:45:37.818415 17982 ts_tablet_manager.cc:1397] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:37.818768 17982 raft_consensus.cc:348] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.818874 17982 raft_consensus.cc:374] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.818910 17982 raft_consensus.cc:729] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46119d7c36344798813f510a02cb05b5, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.819021 17982 consensus_queue.cc:260] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.819262 17982 ts_tablet_manager.cc:1428] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:37.819931 17980 tablet_bootstrap.cc:492] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: No bootstrap required, opened a new log
I20260808 04:45:37.820019 17980 ts_tablet_manager.cc:1397] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260808 04:45:37.820446 17980 raft_consensus.cc:348] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.820554 17980 raft_consensus.cc:374] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:37.820583 17980 raft_consensus.cc:729] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e61140b3a2f4e4fb8eda46f59eb6ab8, State: Initialized, Role: FOLLOWER
I20260808 04:45:37.820703 17980 consensus_queue.cc:260] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.820972 17980 ts_tablet_manager.cc:1428] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260808 04:45:37.941880 18061 raft_consensus.cc:482] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:37.942027 18061 raft_consensus.cc:504] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.942417 18061 leader_election.cc:288] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037), 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163)
I20260808 04:45:37.942876 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266f1be196d8418ba1a105af3fb988c9" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" is_pre_election: true
I20260808 04:45:37.942878 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266f1be196d8418ba1a105af3fb988c9" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46119d7c36344798813f510a02cb05b5" is_pre_election: true
I20260808 04:45:37.943058 17406 raft_consensus.cc:2454] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 0.
I20260808 04:45:37.943059 17854 raft_consensus.cc:2454] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 0.
I20260808 04:45:37.943399 17524 leader_election.cc:302] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46119d7c36344798813f510a02cb05b5, 98a6325ceaec433288922dd72fa4b3db; no voters:
I20260808 04:45:37.943594 18061 raft_consensus.cc:2781] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:37.943655 18061 raft_consensus.cc:482] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:37.943681 18061 raft_consensus.cc:3037] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.944707 18061 raft_consensus.cc:504] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.945081 18061 leader_election.cc:288] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 election: Requested vote from peers 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037), 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163)
I20260808 04:45:37.945463 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266f1be196d8418ba1a105af3fb988c9" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46119d7c36344798813f510a02cb05b5"
I20260808 04:45:37.945461 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266f1be196d8418ba1a105af3fb988c9" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8"
I20260808 04:45:37.945580 17406 raft_consensus.cc:3037] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.945606 17854 raft_consensus.cc:3037] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:37.946520 17406 raft_consensus.cc:2454] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 1.
I20260808 04:45:37.946610 17854 raft_consensus.cc:2454] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 1.
I20260808 04:45:37.946866 17520 leader_election.cc:302] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e61140b3a2f4e4fb8eda46f59eb6ab8, 98a6325ceaec433288922dd72fa4b3db; no voters:
I20260808 04:45:37.947026 18061 raft_consensus.cc:2781] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:37.947167 18061 raft_consensus.cc:686] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 1 LEADER]: Becoming Leader. State: Replica: 98a6325ceaec433288922dd72fa4b3db, State: Running, Role: LEADER
I20260808 04:45:37.947325 18061 consensus_queue.cc:237] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:37.949117 17160 catalog_manager.cc:5495] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 98a6325ceaec433288922dd72fa4b3db (127.4.217.2). New cstate: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:37.970158 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:37.967288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.4.217.2:41421)
0808 04:45:37.967441 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:37.967445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:37.967459 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:45:37.967557 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:37.967559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:37.967602 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:37.967673 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:37.967894 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.968386 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.969263 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:37.969468 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:37.969548 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:37.969764 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:37.969833 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:37.969886 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:37.970008 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 04:45:37.970055 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:45:37.972335 17406 raft_consensus.cc:1261] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Refusing update from remote peer 98a6325ceaec433288922dd72fa4b3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:37.972344 17854 raft_consensus.cc:1261] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Refusing update from remote peer 98a6325ceaec433288922dd72fa4b3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:37.972816 18061 consensus_queue.cc:1035] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:37.972965 17988 consensus_queue.cc:1035] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:37.982685 17159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:37.982829 17159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:37.984683 17159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e: Sending DeleteTablet for 3 replicas of tablet 266f1be196d8418ba1a105af3fb988c9
I20260808 04:45:37.985258 17375 tablet_service.cc:1514] Processing DeleteTablet for tablet 266f1be196d8418ba1a105af3fb988c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:37 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260808 04:45:37.985347 17600 tablet_service.cc:1514] Processing DeleteTablet for tablet 266f1be196d8418ba1a105af3fb988c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:37 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:49110
I20260808 04:45:37.985437 18018 tablet_replica.cc:330] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: stopping tablet replica
I20260808 04:45:37.985531 17822 tablet_service.cc:1514] Processing DeleteTablet for tablet 266f1be196d8418ba1a105af3fb988c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:37 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:41124
I20260808 04:45:37.985554 18018 raft_consensus.cc:2229] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:37.985695 18018 raft_consensus.cc:2258] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:37.985733 18017 tablet_replica.cc:330] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: stopping tablet replica
I20260808 04:45:37.985867 18017 raft_consensus.cc:2229] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:37.986145 18017 raft_consensus.cc:2258] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:37.986339 18019 tablet_replica.cc:330] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: stopping tablet replica
I20260808 04:45:37.986465 18019 raft_consensus.cc:2229] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:37.986598 18019 raft_consensus.cc:2258] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:37.987293 18019 ts_tablet_manager.cc:1905] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:37.987697 18017 ts_tablet_manager.cc:1905] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:37.989048 17159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:45:37.989626 17159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:45:37.989976 18019 ts_tablet_manager.cc:1918] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:37.990069 18019 log.cc:1196] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/266f1be196d8418ba1a105af3fb988c9
I20260808 04:45:37.990216 18017 ts_tablet_manager.cc:1918] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:37.990298 18017 log.cc:1196] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/266f1be196d8418ba1a105af3fb988c9
I20260808 04:45:37.990430 18019 ts_tablet_manager.cc:1939] T 266f1be196d8418ba1a105af3fb988c9 P 46119d7c36344798813f510a02cb05b5: Deleting consensus metadata
I20260808 04:45:37.990563 18017 ts_tablet_manager.cc:1939] T 266f1be196d8418ba1a105af3fb988c9 P 98a6325ceaec433288922dd72fa4b3db: Deleting consensus metadata
I20260808 04:45:37.991098 17143 catalog_manager.cc:4841] TS 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163): tablet 266f1be196d8418ba1a105af3fb988c9 (table test-table [id=6aafea149ff541b7a0a13702adba3c91]) successfully deleted
I20260808 04:45:37.991295 17143 catalog_manager.cc:4841] TS 98a6325ceaec433288922dd72fa4b3db (127.4.217.2:41421): tablet 266f1be196d8418ba1a105af3fb988c9 (table test-table [id=6aafea149ff541b7a0a13702adba3c91]) successfully deleted
I20260808 04:45:37.992552 18018 ts_tablet_manager.cc:1905] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:37.994815 18018 ts_tablet_manager.cc:1918] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:45:37.994882 18018 log.cc:1196] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/266f1be196d8418ba1a105af3fb988c9
I20260808 04:45:37.995172 18018 ts_tablet_manager.cc:1939] T 266f1be196d8418ba1a105af3fb988c9 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Deleting consensus metadata
I20260808 04:45:37.995875 17139 catalog_manager.cc:4841] TS 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037): tablet 266f1be196d8418ba1a105af3fb988c9 (table test-table [id=6aafea149ff541b7a0a13702adba3c91]) successfully deleted
I20260808 04:45:37.997539 17375 tablet_service.cc:1467] Processing CreateTablet for tablet 3fac684b1a43442c84b9f370f99f74e4 (DEFAULT_TABLE table=test-table [id=514f7746dec64366aa11d6c8ca7436be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.997804 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fac684b1a43442c84b9f370f99f74e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:37.997857 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 3fac684b1a43442c84b9f370f99f74e4 (DEFAULT_TABLE table=test-table [id=514f7746dec64366aa11d6c8ca7436be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:37.998107 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fac684b1a43442c84b9f370f99f74e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:38.000003 17980 tablet_bootstrap.cc:492] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Bootstrap starting.
I20260808 04:45:38.000363 17979 tablet_bootstrap.cc:492] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: Bootstrap starting.
I20260808 04:45:38.000571 17822 tablet_service.cc:1467] Processing CreateTablet for tablet 3fac684b1a43442c84b9f370f99f74e4 (DEFAULT_TABLE table=test-table [id=514f7746dec64366aa11d6c8ca7436be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:45:38.001008 17980 tablet_bootstrap.cc:654] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:38.001096 17822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fac684b1a43442c84b9f370f99f74e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:38.001243 17979 tablet_bootstrap.cc:654] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:38.002358 17979 tablet_bootstrap.cc:492] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: No bootstrap required, opened a new log
I20260808 04:45:38.002449 17979 ts_tablet_manager.cc:1397] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:38.002836 17979 raft_consensus.cc:348] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:38.002938 17979 raft_consensus.cc:374] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:38.002967 17979 raft_consensus.cc:729] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a6325ceaec433288922dd72fa4b3db, State: Initialized, Role: FOLLOWER
I20260808 04:45:38.003101 17979 consensus_queue.cc:260] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:38.003393 17979 ts_tablet_manager.cc:1428] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:45:38.003744 17982 tablet_bootstrap.cc:492] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: Bootstrap starting.
I20260808 04:45:38.004423 17982 tablet_bootstrap.cc:654] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:38.005784 17980 tablet_bootstrap.cc:492] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: No bootstrap required, opened a new log
I20260808 04:45:38.005868 17980 ts_tablet_manager.cc:1397] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260808 04:45:38.006114 17982 tablet_bootstrap.cc:492] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: No bootstrap required, opened a new log
I20260808 04:45:38.006204 17982 ts_tablet_manager.cc:1397] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:45:38.006289 17980 raft_consensus.cc:348] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:38.006398 17980 raft_consensus.cc:374] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:38.006436 17980 raft_consensus.cc:729] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e61140b3a2f4e4fb8eda46f59eb6ab8, State: Initialized, Role: FOLLOWER
I20260808 04:45:38.006556 17980 consensus_queue.cc:260] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:38.006624 17982 raft_consensus.cc:348] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:38.006755 17982 raft_consensus.cc:374] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:38.006798 17982 raft_consensus.cc:729] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46119d7c36344798813f510a02cb05b5, State: Initialized, Role: FOLLOWER
I20260808 04:45:38.006933 17982 consensus_queue.cc:260] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:38.007251 17982 ts_tablet_manager.cc:1428] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:45:38.007289 17980 ts_tablet_manager.cc:1428] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:45:38.284653 17988 raft_consensus.cc:482] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:45:38.284850 17988 raft_consensus.cc:504] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:38.285315 17988 leader_election.cc:288] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037), 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163)
I20260808 04:45:38.285838 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fac684b1a43442c84b9f370f99f74e4" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" is_pre_election: true
I20260808 04:45:38.285837 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fac684b1a43442c84b9f370f99f74e4" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46119d7c36344798813f510a02cb05b5" is_pre_election: true
I20260808 04:45:38.285996 17854 raft_consensus.cc:2454] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 0.
I20260808 04:45:38.286007 17406 raft_consensus.cc:2454] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 0.
I20260808 04:45:38.286362 17524 leader_election.cc:302] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46119d7c36344798813f510a02cb05b5, 98a6325ceaec433288922dd72fa4b3db; no voters:
I20260808 04:45:38.286550 17988 raft_consensus.cc:2781] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:45:38.286620 17988 raft_consensus.cc:482] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:45:38.286648 17988 raft_consensus.cc:3037] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:38.287763 17988 raft_consensus.cc:504] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:38.288165 17988 leader_election.cc:288] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 election: Requested vote from peers 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037), 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163)
I20260808 04:45:38.288456 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fac684b1a43442c84b9f370f99f74e4" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8"
I20260808 04:45:38.288604 17406 raft_consensus.cc:3037] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:38.288554 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fac684b1a43442c84b9f370f99f74e4" candidate_uuid: "98a6325ceaec433288922dd72fa4b3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46119d7c36344798813f510a02cb05b5"
I20260808 04:45:38.288734 17854 raft_consensus.cc:3037] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:38.289665 17406 raft_consensus.cc:2454] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 1.
I20260808 04:45:38.289733 17854 raft_consensus.cc:2454] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a6325ceaec433288922dd72fa4b3db in term 1.
I20260808 04:45:38.289974 17520 leader_election.cc:302] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e61140b3a2f4e4fb8eda46f59eb6ab8, 98a6325ceaec433288922dd72fa4b3db; no voters:
I20260808 04:45:38.290169 17988 raft_consensus.cc:2781] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:38.290274 17988 raft_consensus.cc:686] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 1 LEADER]: Becoming Leader. State: Replica: 98a6325ceaec433288922dd72fa4b3db, State: Running, Role: LEADER
I20260808 04:45:38.290422 17988 consensus_queue.cc:237] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } }
I20260808 04:45:38.292023 17158 catalog_manager.cc:5495] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 98a6325ceaec433288922dd72fa4b3db (127.4.217.2). New cstate: current_term: 1 leader_uuid: "98a6325ceaec433288922dd72fa4b3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a6325ceaec433288922dd72fa4b3db" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 } health_report { overall_health: UNKNOWN } } }
I20260808 04:45:38.375269 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:38.372438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:41249 (local address 127.4.217.2:41421)
0808 04:45:38.372564 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:38.372568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:38.372681 (+ 113us) server_negotiation.cc:407] Connection header received
0808 04:45:38.372805 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:38.372807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:38.372847 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:38.372910 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:45:38.373136 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:38.373627 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:38.374421 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:38.374625 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:38.374698 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:38.374897 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:45:38.374960 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:45:38.375013 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:38.375114 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 04:45:38.375153 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 04:45:38.375283 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:38.372386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:41421 (local address 127.4.217.2:41249)
0808 04:45:38.372606 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:38.372619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:38.372707 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:38.372935 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:38.372938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:38.372945 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:45:38.373047 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:38.373052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:38.373712 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:38.373714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:38.374326 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:38.374334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:38.374717 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:38.374720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:38.374780 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:38.374805 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:45:38.375031 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:45:38.375043 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 04:45:38.375092 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 04:45:38.375166 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260808 04:45:38.381207 17406 raft_consensus.cc:1261] T 3fac684b1a43442c84b9f370f99f74e4 P 7e61140b3a2f4e4fb8eda46f59eb6ab8 [term 1 FOLLOWER]: Refusing update from remote peer 98a6325ceaec433288922dd72fa4b3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:38.381313 17854 raft_consensus.cc:1261] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Refusing update from remote peer 98a6325ceaec433288922dd72fa4b3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:45:38.381687 18045 consensus_queue.cc:1035] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e61140b3a2f4e4fb8eda46f59eb6ab8" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:38.381844 17988 consensus_queue.cc:1035] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "46119d7c36344798813f510a02cb05b5" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 46163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:45:38.410741 17158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:38.410895 17158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:45:38.412772 17158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f289c6be5b3b4358b364f70cf8cc221e: Sending DeleteTablet for 3 replicas of tablet 3fac684b1a43442c84b9f370f99f74e4
I20260808 04:45:38.413555 17600 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fac684b1a43442c84b9f370f99f74e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:38 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:49110
I20260808 04:45:38.413556 17375 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fac684b1a43442c84b9f370f99f74e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:38 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260808 04:45:38.413765 17822 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fac684b1a43442c84b9f370f99f74e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:45:38 UTC) from {username='oryx', principal='oryx/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:41124
I20260808 04:45:38.413968 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 17269
I20260808 04:45:38.413965 18017 tablet_replica.cc:330] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: stopping tablet replica
I20260808 04:45:38.414134 18017 raft_consensus.cc:2229] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:45:38.414356 18017 raft_consensus.cc:2258] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:38.418594 18019 tablet_replica.cc:330] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: stopping tablet replica
I20260808 04:45:38.418787 18019 raft_consensus.cc:2229] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:45:38.418949 18019 raft_consensus.cc:2258] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:45:38.423475 18017 ts_tablet_manager.cc:1905] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:38.424525 18019 ts_tablet_manager.cc:1905] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:45:38.426237 18017 ts_tablet_manager.cc:1918] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:38.426299 18017 log.cc:1196] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/3fac684b1a43442c84b9f370f99f74e4
I20260808 04:45:38.426489 18017 ts_tablet_manager.cc:1939] T 3fac684b1a43442c84b9f370f99f74e4 P 98a6325ceaec433288922dd72fa4b3db: Deleting consensus metadata
I20260808 04:45:38.427346 17143 catalog_manager.cc:4841] TS 98a6325ceaec433288922dd72fa4b3db (127.4.217.2:41421): tablet 3fac684b1a43442c84b9f370f99f74e4 (table test-table [id=514f7746dec64366aa11d6c8ca7436be]) successfully deleted
I20260808 04:45:38.427850 18019 ts_tablet_manager.cc:1918] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:45:38.427948 18019 log.cc:1196] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/3fac684b1a43442c84b9f370f99f74e4
I20260808 04:45:38.428241 18019 ts_tablet_manager.cc:1939] T 3fac684b1a43442c84b9f370f99f74e4 P 46119d7c36344798813f510a02cb05b5: Deleting consensus metadata
I20260808 04:45:38.429121 17143 catalog_manager.cc:4841] TS 46119d7c36344798813f510a02cb05b5 (127.4.217.3:46163): tablet 3fac684b1a43442c84b9f370f99f74e4 (table test-table [id=514f7746dec64366aa11d6c8ca7436be]) successfully deleted
W20260808 04:45:38.433175 17139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:35037 (error 108)
I20260808 04:45:38.433295 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 17489
I20260808 04:45:38.443586 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:38.442722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:35037 (local address 127.0.0.1:47916)
0808 04:45:38.443394 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:38.443482 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":424,"threads_started":1}
W20260808 04:45:38.443758 17139 catalog_manager.cc:4584] TS 7e61140b3a2f4e4fb8eda46f59eb6ab8 (127.4.217.1:35037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fac684b1a43442c84b9f370f99f74e4: Network error: Client connection negotiation failed: client connection to 127.4.217.1:35037: connect: Connection refused (error 111)
I20260808 04:45:38.453655 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 17714
I20260808 04:45:38.472365 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 17116
2026-08-08T04:45:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3740 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:38 dist-test-slave-2kcd krb5kdc[18140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:38 dist-test-slave-2kcd krb5kdc[18140](info): set up 2 sockets
Aug 08 04:45:38 dist-test-slave-2kcd krb5kdc[18140](info): commencing operation
krb5kdc: starting...
W20260808 04:45:39.738832 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:39 dist-test-slave-2kcd krb5kdc[18140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:39.862996 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35069
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:39.975251 18161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:39.975503 18161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:39.975580 18161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:39.979513 18161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:39.979568 18161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:39.979586 18161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:39.979652 18161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:39.979681 18161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:39.984930 18161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:39.985930 18161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:39.986955 18161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:39.995561 18172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:39.995618 18178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:39.995541 18173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:39.995990 18161 server_base.cc:1034] running on GCE node
I20260808 04:45:39.996572 18161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:39.997797 18161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:39.998986 18161 hybrid_clock.cc:648] HybridClock initialized: now 1786164339998978 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T04:45:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1539 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:45:40 dist-test-slave-2kcd krb5kdc[18186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:45:40 dist-test-slave-2kcd krb5kdc[18186](info): set up 2 sockets
Aug 08 04:45:40 dist-test-slave-2kcd krb5kdc[18186](info): commencing operation
krb5kdc: starting...
W20260808 04:45:41.294400 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:45:41 dist-test-slave-2kcd krb5kdc[18186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:45:41.419920 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33229
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:33229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:41.531015 18202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:41.531294 18202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:41.531368 18202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:41.535282 18202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:41.535364 18202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:41.535405 18202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:41.535462 18202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:41.535499 18202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:41.540540 18202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:33229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:41.541591 18202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:41.542562 18202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:41.550659 18208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:41.550662 18207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:41.550668 18211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:41.550908 18202 server_base.cc:1034] running on GCE node
I20260808 04:45:41.551435 18202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:41.552413 18202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:41.553581 18202 hybrid_clock.cc:648] HybridClock initialized: now 1786164341553560 us; error 31 us; skew 500 ppm
Aug 08 04:45:41 dist-test-slave-2kcd krb5kdc[18186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164341, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:41.556382 18202 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:41.557294 18202 webserver.cc:460] Webserver started at http://127.4.217.62:45405/ using document root <none> and password file <none>
I20260808 04:45:41.557832 18202 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:41.557894 18202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:41.558135 18202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:41.560079 18202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "e3f0e11fca93444e9201d3664240ed28"
format_stamp: "Formatted at 2026-08-08 04:45:41 on dist-test-slave-2kcd"
server_key: "90831725fdfd1332405c5565b831e430"
server_key_iv: "d471bd7fd5b8d28c2918572f62ec75fa"
server_key_version: "encryptionkey@0"
I20260808 04:45:41.560649 18202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "e3f0e11fca93444e9201d3664240ed28"
format_stamp: "Formatted at 2026-08-08 04:45:41 on dist-test-slave-2kcd"
server_key: "90831725fdfd1332405c5565b831e430"
server_key_iv: "d471bd7fd5b8d28c2918572f62ec75fa"
server_key_version: "encryptionkey@0"
I20260808 04:45:41.564157 18202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:45:41.566501 18220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:41.567660 18202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260808 04:45:41.567785 18202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "e3f0e11fca93444e9201d3664240ed28"
format_stamp: "Formatted at 2026-08-08 04:45:41 on dist-test-slave-2kcd"
server_key: "90831725fdfd1332405c5565b831e430"
server_key_iv: "d471bd7fd5b8d28c2918572f62ec75fa"
server_key_version: "encryptionkey@0"
I20260808 04:45:41.567883 18202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:41.586362 18202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:41.589242 18202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:41.589444 18202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:41.596614 18202 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:33229
I20260808 04:45:41.596676 18307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:33229 every 8 connection(s)
I20260808 04:45:41.597621 18202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:41.600441 18308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:41.606555 18308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: Bootstrap starting.
I20260808 04:45:41.607052 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 18202
I20260808 04:45:41.607173 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:41.607445 4964 external_mini_cluster.cc:1428] Setting key baa93d0fd7d739186a767f4f921bce1a
I20260808 04:45:41.609387 18308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:41.610309 18308 log.cc:824] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:41.612208 18308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: No bootstrap required, opened a new log
I20260808 04:45:41.615968 18308 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } }
I20260808 04:45:41.616226 18308 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:41.616312 18308 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0e11fca93444e9201d3664240ed28, State: Initialized, Role: FOLLOWER
Aug 08 04:45:41 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:41.616850 18308 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } }
I20260808 04:45:41.617029 18308 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:41.617130 18308 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:41.617231 18308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:41.618392 18308 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } }
I20260808 04:45:41.618757 18308 leader_election.cc:302] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3f0e11fca93444e9201d3664240ed28; no voters:
I20260808 04:45:41.619074 18308 leader_election.cc:288] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:41.619220 18316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:41.619463 18316 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 1 LEADER]: Becoming Leader. State: Replica: e3f0e11fca93444e9201d3664240ed28, State: Running, Role: LEADER
I20260808 04:45:41.619867 18316 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } }
I20260808 04:45:41.620383 18308 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:41.621814 18318 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3f0e11fca93444e9201d3664240ed28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } } }
I20260808 04:45:41.621945 18318 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:41.622330 18330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:41.621826 18320 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f0e11fca93444e9201d3664240ed28. Latest consensus state: current_term: 1 leader_uuid: "e3f0e11fca93444e9201d3664240ed28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } } }
I20260808 04:45:41.622465 18320 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:41.623484 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:41.608868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.4.217.62:33229)
0808 04:45:41.609335 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:41.609357 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:41.609403 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:45:41.610303 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:41.610329 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:41.610774 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:41.611120 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:41.611969 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:41.613128 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:41.613979 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:41.614461 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:41.614643 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:41.617375 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:41.617380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:41.617392 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:41.617420 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:41.619253 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:41.619830 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:41.619836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:41.619842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:41.619994 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:41.620442 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:41.620445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:41.620447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:41.620996 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:41.621215 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:41.622195 (+ 980us) server_negotiation.cc:299] Negotiation successful
0808 04:45:41.622527 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":122,"threads_started":1}
I20260808 04:45:41.625541 18330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:41.631009 18330 catalog_manager.cc:1347] Generated new cluster ID: 8e50db81c0994c2490b71ee44ca0f917
I20260808 04:45:41.631073 18330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:41.638612 18330 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:41.639833 18330 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:41.649748 18330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: Generated new TSK 0
I20260808 04:45:41.650621 18330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:45:41.686045 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33229
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:41.804229 18351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:41.804502 18351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:41.804591 18351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:41.808578 18351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:41.808661 18351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:41.808753 18351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:41.813836 18351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:41.814843 18351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:41.815837 18351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:41.825403 18363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:41.825397 18357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:41.825361 18358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:41.825623 18351 server_base.cc:1034] running on GCE node
I20260808 04:45:41.825999 18351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:41.826651 18351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:41.827850 18351 hybrid_clock.cc:648] HybridClock initialized: now 1786164341827825 us; error 32 us; skew 500 ppm
Aug 08 04:45:41 dist-test-slave-2kcd krb5kdc[18186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164341, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:41.830688 18351 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:45:41.831494 18351 webserver.cc:460] Webserver started at http://127.4.217.1:37645/ using document root <none> and password file <none>
I20260808 04:45:41.832053 18351 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:41.832150 18351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:41.832379 18351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:41.834357 18351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "0ae4e2b315424a728b1c8e7db07eb025"
format_stamp: "Formatted at 2026-08-08 04:45:41 on dist-test-slave-2kcd"
server_key: "675c063cae3894ef596fa77376ce3485"
server_key_iv: "b2edd9a5e196f22bb628b87568ea7ab4"
server_key_version: "encryptionkey@0"
I20260808 04:45:41.834926 18351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "0ae4e2b315424a728b1c8e7db07eb025"
format_stamp: "Formatted at 2026-08-08 04:45:41 on dist-test-slave-2kcd"
server_key: "675c063cae3894ef596fa77376ce3485"
server_key_iv: "b2edd9a5e196f22bb628b87568ea7ab4"
server_key_version: "encryptionkey@0"
I20260808 04:45:41.838572 18351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:45:41.840978 18377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:41.842202 18351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:41.842320 18351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "0ae4e2b315424a728b1c8e7db07eb025"
format_stamp: "Formatted at 2026-08-08 04:45:41 on dist-test-slave-2kcd"
server_key: "675c063cae3894ef596fa77376ce3485"
server_key_iv: "b2edd9a5e196f22bb628b87568ea7ab4"
server_key_version: "encryptionkey@0"
I20260808 04:45:41.842438 18351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:41.888525 18351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:41.891932 18351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:41.892168 18351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:41.892864 18351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:41.893756 18351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:41.893807 18351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:41.893880 18351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:41.893919 18351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:41.903343 18351 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:34631
I20260808 04:45:41.903403 18554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:34631 every 8 connection(s)
I20260808 04:45:41.904249 18351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
Aug 08 04:45:41 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164341, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:41.914314 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 18351
I20260808 04:45:41.914461 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:41.914836 4964 external_mini_cluster.cc:1428] Setting key 4d762c168412bec573458d595ce41eaf
I20260808 04:45:41.917927 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:41.906081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:43357 (local address 127.4.217.62:33229)
0808 04:45:41.906217 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:41.906221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:41.907180 (+ 959us) server_negotiation.cc:407] Connection header received
0808 04:45:41.908115 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:41.908119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:41.908184 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:41.908324 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:41.909591 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:41.910130 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:41.911286 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:41.911538 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:41.911656 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:41.914037 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:41.914040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:41.914043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:41.914074 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:41.915615 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:41.916201 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:41.916208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:41.916213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:41.916310 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:41.916694 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:41.916698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:41.916699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:41.916960 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:41.917078 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:41.917541 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 04:45:41.917683 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:45:41.918506 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:41.906359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:43357)
0808 04:45:41.907024 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:41.907070 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:41.907897 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:41.908506 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:41.908513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:41.908962 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:41.909435 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:41.909449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:41.910264 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:41.910267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:41.911151 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:41.911161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:41.911708 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:41.911713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:41.911877 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:41.912378 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:41.912394 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:41.913863 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:41.915732 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:41.915738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:41.915752 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:41.916075 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:41.916415 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:41.916418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:41.916420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:41.916586 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:41.917084 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:41.917089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:41.917300 (+ 211us) client_negotiation.cc:769] Sending connection context
0808 04:45:41.917451 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 04:45:41.917701 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":125,"threads_started":1}
I20260808 04:45:41.919939 18556 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33229
I20260808 04:45:41.920194 18556 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:41.920722 18556 heartbeater.cc:511] Master 127.4.217.62:33229 requested a full tablet report, sending...
I20260808 04:45:41.922397 18250 ts_manager.cc:194] Registered new tserver with Master: 0ae4e2b315424a728b1c8e7db07eb025 (127.4.217.1:34631)
I20260808 04:45:41.923599 18250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:43357
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:45:41.943892 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33229
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:42.056885 18567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:42.057193 18567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:42.057273 18567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:42.061254 18567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:42.061352 18567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:42.061445 18567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:42.066679 18567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:42.068059 18567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:42.069253 18567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:42.079440 18573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:42.079427 18577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:42.079514 18574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:42.079942 18567 server_base.cc:1034] running on GCE node
I20260808 04:45:42.080381 18567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:42.081139 18567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:42.082304 18567 hybrid_clock.cc:648] HybridClock initialized: now 1786164342082269 us; error 47 us; skew 500 ppm
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164342, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:42.085232 18567 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:45:42.086220 18567 webserver.cc:460] Webserver started at http://127.4.217.2:41459/ using document root <none> and password file <none>
I20260808 04:45:42.086803 18567 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:42.086869 18567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:42.087052 18567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:42.089182 18567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "83f989aa6b804e36bcb51aac3304d609"
format_stamp: "Formatted at 2026-08-08 04:45:42 on dist-test-slave-2kcd"
server_key: "2e4ac2d36fe74029f41be553f62d3813"
server_key_iv: "4afc7abb6f76b0a9ed0df155dfd9e9c5"
server_key_version: "encryptionkey@0"
I20260808 04:45:42.089767 18567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "83f989aa6b804e36bcb51aac3304d609"
format_stamp: "Formatted at 2026-08-08 04:45:42 on dist-test-slave-2kcd"
server_key: "2e4ac2d36fe74029f41be553f62d3813"
server_key_iv: "4afc7abb6f76b0a9ed0df155dfd9e9c5"
server_key_version: "encryptionkey@0"
I20260808 04:45:42.093605 18567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:45:42.096140 18592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.097427 18567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:42.097563 18567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "83f989aa6b804e36bcb51aac3304d609"
format_stamp: "Formatted at 2026-08-08 04:45:42 on dist-test-slave-2kcd"
server_key: "2e4ac2d36fe74029f41be553f62d3813"
server_key_iv: "4afc7abb6f76b0a9ed0df155dfd9e9c5"
server_key_version: "encryptionkey@0"
I20260808 04:45:42.097677 18567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:42.128670 18567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:42.131790 18567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:42.132016 18567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:42.132668 18567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:42.133590 18567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:42.133641 18567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.133711 18567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:42.133751 18567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.143141 18567 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:45869
I20260808 04:45:42.143174 18787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:45869 every 8 connection(s)
I20260808 04:45:42.144124 18567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:42.150671 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 18567
I20260808 04:45:42.150779 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:42.151091 4964 external_mini_cluster.cc:1428] Setting key 0460e8f945cd6a03de31cf79dc071239
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164342, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:42.158494 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:42.146129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:51077 (local address 127.4.217.62:33229)
0808 04:45:42.146288 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:42.146292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:42.147395 (+ 1103us) server_negotiation.cc:407] Connection header received
0808 04:45:42.148358 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:42.148361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:42.148427 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:42.148547 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:42.149882 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.150470 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.151893 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.152114 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.152230 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.154738 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:42.154742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:42.154744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:42.154775 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:42.156293 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:42.156876 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:42.156881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:42.156883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:42.156966 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:42.157333 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:42.157337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:42.157338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:42.157526 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:42.157647 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:42.158189 (+ 542us) server_negotiation.cc:299] Negotiation successful
0808 04:45:42.158315 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:45:42.159319 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:42.146566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:51077)
0808 04:45:42.147226 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:42.147266 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:42.148130 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:42.148740 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:42.148747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:42.149159 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:42.149721 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.149737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.150617 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.150621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.151758 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.151769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.152273 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.152277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.152444 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.153112 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:42.153129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:42.154569 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:42.156423 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:42.156431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:42.156446 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:42.156714 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:42.157072 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:42.157076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:42.157078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:42.157223 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:42.157645 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:42.157653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:42.157903 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 04:45:42.158129 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 04:45:42.158458 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":104,"threads_started":1}
I20260808 04:45:42.160719 18788 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33229
I20260808 04:45:42.161020 18788 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:42.161540 18788 heartbeater.cc:511] Master 127.4.217.62:33229 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:45:42.162827 18250 ts_manager.cc:194] Registered new tserver with Master: 83f989aa6b804e36bcb51aac3304d609 (127.4.217.2:45869)
I20260808 04:45:42.163489 18250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:51077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:45:42.181587 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33229
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:42.297694 18803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:42.297914 18803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:42.297989 18803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:42.301970 18803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:42.302021 18803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:42.302124 18803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:42.307237 18803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:42.308193 18803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:42.309190 18803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:42.318305 18822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:42.318310 18815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:42.318305 18816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:42.318701 18803 server_base.cc:1034] running on GCE node
I20260808 04:45:42.319134 18803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:42.319710 18803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:42.320855 18803 hybrid_clock.cc:648] HybridClock initialized: now 1786164342320811 us; error 54 us; skew 500 ppm
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164342, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:42.323711 18803 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:45:42.324448 18803 webserver.cc:460] Webserver started at http://127.4.217.3:33451/ using document root <none> and password file <none>
I20260808 04:45:42.324990 18803 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:42.325058 18803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:42.325214 18803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:42.327018 18803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "5dc13ac8d8c6492bb02ecb2e1480095e"
format_stamp: "Formatted at 2026-08-08 04:45:42 on dist-test-slave-2kcd"
server_key: "068b48fa1b07554e07c68e109b4b67dd"
server_key_iv: "eee24c9e6d165a20a00c0f5d43ffd739"
server_key_version: "encryptionkey@0"
I20260808 04:45:42.327510 18803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "5dc13ac8d8c6492bb02ecb2e1480095e"
format_stamp: "Formatted at 2026-08-08 04:45:42 on dist-test-slave-2kcd"
server_key: "068b48fa1b07554e07c68e109b4b67dd"
server_key_iv: "eee24c9e6d165a20a00c0f5d43ffd739"
server_key_version: "encryptionkey@0"
I20260808 04:45:42.331017 18803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 04:45:42.333299 18832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.334388 18803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:42.334511 18803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "5dc13ac8d8c6492bb02ecb2e1480095e"
format_stamp: "Formatted at 2026-08-08 04:45:42 on dist-test-slave-2kcd"
server_key: "068b48fa1b07554e07c68e109b4b67dd"
server_key_iv: "eee24c9e6d165a20a00c0f5d43ffd739"
server_key_version: "encryptionkey@0"
I20260808 04:45:42.334614 18803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:42.358021 18803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:42.361271 18803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:42.361469 18803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:42.362051 18803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:42.362843 18803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:42.362900 18803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.362941 18803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:42.362985 18803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.374029 18803 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:39119
I20260808 04:45:42.374068 19014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:39119 every 8 connection(s)
I20260808 04:45:42.374879 18803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:42.378386 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 18803
I20260808 04:45:42.378491 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:42.378832 4964 external_mini_cluster.cc:1428] Setting key 2ca162d0312d7f642deca43ab1614df7
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164342, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:45:42.388077 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:42.376582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:44679 (local address 127.4.217.62:33229)
0808 04:45:42.376754 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:42.376759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:42.377790 (+ 1031us) server_negotiation.cc:407] Connection header received
0808 04:45:42.378690 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:42.378694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:42.378758 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:42.378858 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:42.380273 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.381089 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.382217 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.382421 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.382499 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.384742 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:42.384745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:42.384747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:42.384769 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:42.386225 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:42.386680 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:42.386683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:42.386684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:42.386753 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:42.387179 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:42.387181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:42.387182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:42.387347 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:42.387425 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:42.387863 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 04:45:42.387951 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:45:42.388882 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:42.376985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:44679)
0808 04:45:42.377641 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:42.377681 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:42.378413 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:42.379057 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:42.379063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:42.379484 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:42.380091 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.380128 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.381239 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.381243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.382109 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.382120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.382551 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.382554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.382693 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.383203 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:42.383220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:42.384593 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:42.386327 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:42.386332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:42.386343 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:42.386593 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:42.386913 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:45:42.386916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:45:42.386917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:45:42.387089 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:45:42.387439 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:42.387444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:42.387663 (+ 219us) client_negotiation.cc:769] Sending connection context
0808 04:45:42.387812 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:45:42.388042 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":112,"threads_started":1}
I20260808 04:45:42.389984 19016 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33229
I20260808 04:45:42.390218 19016 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:42.390689 19016 heartbeater.cc:511] Master 127.4.217.62:33229 requested a full tablet report, sending...
I20260808 04:45:42.391857 18250 ts_manager.cc:194] Registered new tserver with Master: 5dc13ac8d8c6492bb02ecb2e1480095e (127.4.217.3:39119)
I20260808 04:45:42.392611 18250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:44679
I20260808 04:45:42.393018 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab.
Entry for principal oryx/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab.
WARNING: no policy specified for oryx/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab.
Entry for principal oryx/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab.
WARNING: no policy specified for oryx/127.4.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab.
Entry for principal oryx/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab.
I20260808 04:45:42.473287 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 18351
I20260808 04:45:42.488967 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 18567
I20260808 04:45:42.507771 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 18803
I20260808 04:45:42.524065 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 18202
I20260808 04:45:42.537797 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33229
--webserver_interface=127.4.217.62
--webserver_port=45405
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:33229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:45:42.650380 19033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:42.650660 19033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:42.650732 19033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:42.654700 19033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:42.654775 19033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:42.654814 19033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:42.654867 19033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:42.654906 19033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:42.659957 19033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:33229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=45405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:42.661029 19033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:42.662034 19033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:42.670503 19043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:42.670610 19039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:42.670492 19040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:42.670702 19033 server_base.cc:1034] running on GCE node
I20260808 04:45:42.671324 19033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:42.672379 19033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:42.673542 19033 hybrid_clock.cc:648] HybridClock initialized: now 1786164342673526 us; error 26 us; skew 500 ppm
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164342, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:42.676196 19033 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:45:42.677147 19033 webserver.cc:460] Webserver started at http://127.4.217.62:45405/ using document root <none> and password file <none>
I20260808 04:45:42.677692 19033 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:42.677798 19033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:42.681531 19033 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:42.683507 19055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.684702 19033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:42.684876 19033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "e3f0e11fca93444e9201d3664240ed28"
format_stamp: "Formatted at 2026-08-08 04:45:41 on dist-test-slave-2kcd"
server_key: "90831725fdfd1332405c5565b831e430"
server_key_iv: "d471bd7fd5b8d28c2918572f62ec75fa"
server_key_version: "encryptionkey@0"
I20260808 04:45:42.685395 19033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:42.701896 19033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:42.704457 19033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:42.704648 19033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:42.711576 19033 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:33229
I20260808 04:45:42.711594 19131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:33229 every 8 connection(s)
I20260808 04:45:42.712567 19033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:42.714221 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 19033
I20260808 04:45:42.714706 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:34631
--local_ip_for_outbound_sockets=127.4.217.1
--tserver_master_addrs=127.4.217.62:33229
--webserver_port=37645
--webserver_interface=127.4.217.1
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.217.1
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:45:42.717546 19132 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:45:42.719137 19132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: Bootstrap starting.
I20260808 04:45:42.736478 19132 log.cc:824] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:42.743017 19132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:45:42.743392 19132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: Bootstrap complete.
I20260808 04:45:42.747578 19132 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } }
I20260808 04:45:42.747960 19132 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0e11fca93444e9201d3664240ed28, State: Initialized, Role: FOLLOWER
I20260808 04:45:42.748441 19132 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } }
I20260808 04:45:42.748603 19132 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:42.748675 19132 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:42.748807 19132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:45:42.750748 19132 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } }
I20260808 04:45:42.751085 19132 leader_election.cc:302] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3f0e11fca93444e9201d3664240ed28; no voters:
I20260808 04:45:42.751502 19132 leader_election.cc:288] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:45:42.751592 19140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:45:42.751806 19140 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [term 2 LEADER]: Becoming Leader. State: Replica: e3f0e11fca93444e9201d3664240ed28, State: Running, Role: LEADER
I20260808 04:45:42.752095 19140 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } }
I20260808 04:45:42.752753 19132 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:42.753044 19140 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f0e11fca93444e9201d3664240ed28. Latest consensus state: current_term: 2 leader_uuid: "e3f0e11fca93444e9201d3664240ed28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } } }
I20260808 04:45:42.753150 19140 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:42.754350 19144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:42.754995 19141 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e3f0e11fca93444e9201d3664240ed28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f0e11fca93444e9201d3664240ed28" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33229 } } }
I20260808 04:45:42.755745 19141 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:42.757596 19144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:42.758235 19144 catalog_manager.cc:1259] Loaded cluster ID: 8e50db81c0994c2490b71ee44ca0f917
I20260808 04:45:42.758299 19144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:42.760658 19144 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:42.761348 19144 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e3f0e11fca93444e9201d3664240ed28: Loaded TSK: 0
I20260808 04:45:42.762574 19144 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:45:42.831521 19136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:42.831812 19136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:42.831887 19136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:42.835958 19136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:42.836062 19136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:42.836158 19136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:42.841616 19136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.1.keytab
--principal=oryx/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:34631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=37645
--tserver_master_addrs=127.4.217.62:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:42.842671 19136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:42.843763 19136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:42.853744 19186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:42.853765 19182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:42.853762 19184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:42.854795 19136 server_base.cc:1034] running on GCE node
I20260808 04:45:42.855230 19136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:42.855988 19136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:42.857172 19136 hybrid_clock.cc:648] HybridClock initialized: now 1786164342857144 us; error 46 us; skew 500 ppm
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:42.860087 19136 init.cc:377] Logged in from keytab as oryx/127.4.217.1@KRBTEST.COM (short username oryx)
I20260808 04:45:42.861045 19136 webserver.cc:460] Webserver started at http://127.4.217.1:37645/ using document root <none> and password file <none>
I20260808 04:45:42.861691 19136 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:42.861800 19136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:42.865821 19136 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:42.868048 19199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.869427 19136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:45:42.869572 19136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "0ae4e2b315424a728b1c8e7db07eb025"
format_stamp: "Formatted at 2026-08-08 04:45:41 on dist-test-slave-2kcd"
server_key: "675c063cae3894ef596fa77376ce3485"
server_key_iv: "b2edd9a5e196f22bb628b87568ea7ab4"
server_key_version: "encryptionkey@0"
I20260808 04:45:42.870118 19136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:42.893172 19136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:42.896466 19136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:42.896692 19136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:42.897358 19136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:42.898252 19136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:42.898325 19136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.898396 19136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:42.898443 19136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:42.907121 19136 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:34631
I20260808 04:45:42.907187 19383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:34631 every 8 connection(s)
I20260808 04:45:42.908152 19136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:42.912371 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 19136
I20260808 04:45:42.912849 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:45869
--local_ip_for_outbound_sockets=127.4.217.2
--tserver_master_addrs=127.4.217.62:33229
--webserver_port=41459
--webserver_interface=127.4.217.2
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.217.2
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:45:42.921118 19389 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:42.910299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:33745 (local address 127.4.217.62:33229)
0808 04:45:42.910861 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:42.910888 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:42.911308 (+ 420us) server_negotiation.cc:407] Connection header received
0808 04:45:42.912387 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:42.912397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:42.912733 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:42.913070 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:42.914214 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.915184 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.916320 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.916832 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.917018 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.920365 (+ 3347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:33745: BlockingRecv error: recv got EOF from 127.4.217.1:33745 (error 108)
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":144,"threads_started":1}
I20260808 04:45:42.921123 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:42.910425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:33745)
0808 04:45:42.911149 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:42.911189 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:42.911981 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:42.913158 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:42.913166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:42.913584 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:42.914026 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.914048 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.915322 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.915331 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.916186 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.916196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.917002 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.917006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.917198 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.917787 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:42.917804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:42.920149 (+ 2345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":132,"threads_started":1}
W20260808 04:45:42.921594 19387 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:33229 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.217.62:33229: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:42.927045 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:42.922092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:46749)
0808 04:45:42.922227 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:42.922240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:42.922304 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:42.922600 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:42.922603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:42.922757 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:42.922842 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.922847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.923554 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.923560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.924118 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.924126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.924564 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.924568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.924661 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.925151 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:42.925162 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:42.926880 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:45:42.927163 19389 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:42.922164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:46749 (local address 127.4.217.62:33229)
0808 04:45:42.922418 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:42.922421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:42.922431 (+ 10us) server_negotiation.cc:407] Connection header received
0808 04:45:42.922460 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:42.922462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:42.922495 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:42.922583 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:42.922959 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.923437 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.924241 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.924448 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.924558 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.927048 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:46749: BlockingRecv error: recv got EOF from 127.4.217.1:46749 (error 108)
Metrics: {"server-negotiator.queue_time_us":192}
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:42 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:42.933346 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:42.928451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:42951)
0808 04:45:42.928677 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:42.928690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:42.928801 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:42.929029 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:42.929040 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:42.929192 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:42.929257 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.929262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.929959 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.929964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.930500 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:42.930508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.930951 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.930954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:42.931040 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.931502 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:42.931524 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:42.933196 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158}
W20260808 04:45:42.933426 19389 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:42.928539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:42951 (local address 127.4.217.62:33229)
0808 04:45:42.928656 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:42.928660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:42.928752 (+ 92us) server_negotiation.cc:407] Connection header received
0808 04:45:42.928880 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:42.928883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:42.928929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:42.929014 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:42.929370 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.929849 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.930627 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:42.930836 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:42.930938 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:42.933313 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:42951: BlockingRecv error: recv got EOF from 127.4.217.1:42951 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 04:45:42.933807 19387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:45:43.030439 19391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:43.030675 19391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:43.030716 19391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:43.034567 19391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:43.034616 19391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:43.034670 19391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:43.039710 19391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.2.keytab
--principal=oryx/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:45869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=41459
--tserver_master_addrs=127.4.217.62:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:43.040619 19391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:43.041563 19391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:43.050848 19396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:43.050853 19398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:43.051061 19391 server_base.cc:1034] running on GCE node
W20260808 04:45:43.050988 19400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:43.051534 19391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:43.052127 19391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:43.053288 19391 hybrid_clock.cc:648] HybridClock initialized: now 1786164343053274 us; error 24 us; skew 500 ppm
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164343, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:43.056008 19391 init.cc:377] Logged in from keytab as oryx/127.4.217.2@KRBTEST.COM (short username oryx)
I20260808 04:45:43.056932 19391 webserver.cc:460] Webserver started at http://127.4.217.2:41459/ using document root <none> and password file <none>
I20260808 04:45:43.057468 19391 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:43.057570 19391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:43.061291 19391 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 04:45:43.063262 19415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:43.064432 19391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:43.064568 19391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "83f989aa6b804e36bcb51aac3304d609"
format_stamp: "Formatted at 2026-08-08 04:45:42 on dist-test-slave-2kcd"
server_key: "2e4ac2d36fe74029f41be553f62d3813"
server_key_iv: "4afc7abb6f76b0a9ed0df155dfd9e9c5"
server_key_version: "encryptionkey@0"
I20260808 04:45:43.065119 19391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:43.081779 19391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:43.084465 19391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:43.084673 19391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:43.085278 19391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:43.086090 19391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:43.086139 19391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:43.086208 19391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:43.086251 19391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:43.095626 19391 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:45869
I20260808 04:45:43.095651 19592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:45869 every 8 connection(s)
I20260808 04:45:43.096582 19391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:43.100472 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 19391
I20260808 04:45:43.101023 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:39119
--local_ip_for_outbound_sockets=127.4.217.3
--tserver_master_addrs=127.4.217.62:33229
--webserver_port=33451
--webserver_interface=127.4.217.3
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.217.3
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:45:43.107646 19389 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:43.098355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:38707 (local address 127.4.217.62:33229)
0808 04:45:43.098491 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:43.098495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:43.099505 (+ 1010us) server_negotiation.cc:407] Connection header received
0808 04:45:43.100491 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:43.100494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:43.100576 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:43.100668 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:43.102239 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.102756 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.103905 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.104138 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.104255 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.107522 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:38707: BlockingRecv error: recv got EOF from 127.4.217.2:38707 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:45:43.108228 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:43.098727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:38707)
0808 04:45:43.099362 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:43.099402 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:43.100236 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:43.100872 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:43.100881 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:43.101387 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:43.102071 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.102093 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.102896 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.102900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.103763 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.103774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.104298 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.104302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.104473 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.105097 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:43.105113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:43.107406 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":100,"threads_started":1}
W20260808 04:45:43.108711 19594 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:33229 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.217.62:33229: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:43.114073 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:43.109234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:56631)
0808 04:45:43.109364 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:43.109380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:43.109457 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:43.109727 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:43.109730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:43.109881 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:43.109965 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.109970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.110664 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.110669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.111208 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.111216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.111653 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.111656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.111751 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.112235 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:43.112246 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:43.113939 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:45:43.114182 19389 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:43.109313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:56631 (local address 127.4.217.62:33229)
0808 04:45:43.109431 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:43.109435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:43.109448 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:43.109585 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:43.109593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:43.109631 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:43.109710 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:43.110081 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.110555 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.111331 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.111542 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.111650 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.114056 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:56631: BlockingRecv error: recv got EOF from 127.4.217.2:56631 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:43.119596 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:43.114632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:52413)
0808 04:45:43.114779 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:43.114791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:43.114855 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:43.115074 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:43.115077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:43.115230 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:43.115312 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.115318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.115966 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.115969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.116841 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.116852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.117243 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.117247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.117330 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.117716 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:43.117729 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:43.119461 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260808 04:45:43.119668 19389 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:43.114715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:52413 (local address 127.4.217.62:33229)
0808 04:45:43.114820 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:43.114823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:43.114835 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:45:43.114932 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:43.114935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:43.114965 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:43.115029 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:43.115420 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.115869 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.116950 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.117141 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.117218 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.119564 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:52413: BlockingRecv error: recv got EOF from 127.4.217.2:52413 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 04:45:43.119993 19594 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:45:43.220233 19598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:43.220499 19598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:43.220543 19598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:43.224499 19598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:43.224556 19598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:43.224630 19598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:43.229969 19598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/krb5kdc/oryx_127.4.217.3.keytab
--principal=oryx/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:39119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=33451
--tserver_master_addrs=127.4.217.62:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:43.230993 19598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:43.232034 19598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:43.241909 19605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:43.242053 19608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:43.241931 19604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:43.242344 19598 server_base.cc:1034] running on GCE node
I20260808 04:45:43.242784 19598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:43.243433 19598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:43.244634 19598 hybrid_clock.cc:648] HybridClock initialized: now 1786164343244598 us; error 46 us; skew 500 ppm
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164343, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:45:43.247967 19598 init.cc:377] Logged in from keytab as oryx/127.4.217.3@KRBTEST.COM (short username oryx)
I20260808 04:45:43.249003 19598 webserver.cc:460] Webserver started at http://127.4.217.3:33451/ using document root <none> and password file <none>
I20260808 04:45:43.249632 19598 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:43.249732 19598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:43.253628 19598 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:45:43.255832 19620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:43.257148 19598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:45:43.257282 19598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "5dc13ac8d8c6492bb02ecb2e1480095e"
format_stamp: "Formatted at 2026-08-08 04:45:42 on dist-test-slave-2kcd"
server_key: "068b48fa1b07554e07c68e109b4b67dd"
server_key_iv: "eee24c9e6d165a20a00c0f5d43ffd739"
server_key_version: "encryptionkey@0"
I20260808 04:45:43.257799 19598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:43.274771 19598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:43.277787 19598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:43.278045 19598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:43.278730 19598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:43.279651 19598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:43.279700 19598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:43.279770 19598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:43.279809 19598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:43.289258 19598 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:39119
I20260808 04:45:43.289295 19785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:39119 every 8 connection(s)
I20260808 04:45:43.290349 19598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:43.298843 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 19598
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:45:43.301838 19389 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:43.292287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:42655 (local address 127.4.217.62:33229)
0808 04:45:43.292440 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:43.292444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:43.293615 (+ 1171us) server_negotiation.cc:407] Connection header received
0808 04:45:43.294464 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:43.294466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:43.294515 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:43.294595 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:43.295803 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.296313 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.297473 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.297695 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.297777 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.301736 (+ 3959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:42655: BlockingRecv error: recv got EOF from 127.4.217.3:42655 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:45:43.302611 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:43.292701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:42655)
0808 04:45:43.293450 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:43.293492 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:43.294277 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:43.294784 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:43.294790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:43.295170 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:43.295664 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.295677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.296433 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.296436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.297361 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.297371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.297840 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.297843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.297990 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.298665 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:43.298682 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:43.301623 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":116,"threads_started":1}
W20260808 04:45:43.303120 19786 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:33229 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.217.62:33229: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:43.309628 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:43.303596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:50593)
0808 04:45:43.303731 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:43.303747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:43.303817 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:43.304072 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:43.304074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:43.304263 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:43.304395 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.304402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.305479 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.305482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.306309 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.306321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.307144 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.307148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.307241 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.307618 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:43.307638 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:43.309485 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 04:45:43.309763 19389 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:43.303706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:50593 (local address 127.4.217.62:33229)
0808 04:45:43.303814 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:43.303818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:43.303831 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:43.303902 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:43.303904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:43.303949 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:43.304039 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:43.304528 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.305354 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.306434 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.306784 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.306869 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.309634 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:50593: BlockingRecv error: recv got EOF from 127.4.217.3:50593 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 04:45:43.310182 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:43.299986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.4.217.62:33229)
0808 04:45:43.300270 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:43.300285 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:43.300301 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:45:43.300437 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:43.300440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:43.300491 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:43.300600 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:43.301456 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.302226 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.303378 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.303693 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.303800 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.305554 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:43.305561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:43.305582 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:45:43.305614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:43.307785 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:43.308252 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:43.308257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:43.308262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:43.308389 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:45:43.308698 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:45:43.308701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:45:43.308705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:45:43.309196 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:43.309410 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:43.309742 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 04:45:43.310010 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:43.315382 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:43.310308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:46249)
0808 04:45:43.310660 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:43.310671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:43.310771 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:43.311021 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:43.311024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:43.311210 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:43.311296 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.311303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.312061 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.312064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.312711 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.312723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.313293 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.313295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.313360 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.313717 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:43.313728 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:43.315290 (+ 1562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
W20260808 04:45:43.315517 19794 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:43.310403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:46249 (local address 127.4.217.62:33229)
0808 04:45:43.310569 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:43.310573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:43.310736 (+ 163us) server_negotiation.cc:407] Connection header received
0808 04:45:43.310875 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:43.310878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:43.310923 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:43.311004 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:43.311418 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.311914 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.312863 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.313178 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.313270 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.315410 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:46249: BlockingRecv error: recv got EOF from 127.4.217.3:46249 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260808 04:45:43.315822 19786 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:43 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:43.941061 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:43.934454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:52091)
0808 04:45:43.934798 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:43.934817 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:43.934971 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:43.935295 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:43.935298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:43.935525 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:43.935644 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.935650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.936460 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.936464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.937422 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:43.937433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.937921 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.937925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:43.938013 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.938532 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:43.938550 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:43.940860 (+ 2310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
W20260808 04:45:43.941419 19804 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:43.934580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:52091 (local address 127.4.217.62:33229)
0808 04:45:43.934830 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:43.934834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:43.934907 (+ 73us) server_negotiation.cc:407] Connection header received
0808 04:45:43.935101 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:43.935104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:43.935156 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:43.935239 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:43.935779 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.936345 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.937559 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:43.937787 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:43.937895 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:43.941017 (+ 3122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:52091: BlockingRecv error: recv got EOF from 127.4.217.1:52091 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
Aug 08 04:45:44 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:44 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:44.125716 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:44.120652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:51345)
0808 04:45:44.120941 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:44.120955 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:44.121062 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:44.121283 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:44.121285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:44.121454 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:44.121550 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:44.121555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.122269 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.122272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:44.122830 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:44.122837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.123245 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.123247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:44.123309 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:44.123790 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:44.123801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:44.125590 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
W20260808 04:45:44.125811 19804 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:44.120714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:51345 (local address 127.4.217.62:33229)
0808 04:45:44.120904 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:44.120908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:44.121004 (+ 96us) server_negotiation.cc:407] Connection header received
0808 04:45:44.121133 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:44.121135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:44.121176 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:44.121242 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:44.121654 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.122168 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.122934 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.123146 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.123223 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:44.125722 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:51345: BlockingRecv error: recv got EOF from 127.4.217.2:51345 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 08 04:45:44 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:44 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:44.322285 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:44.316500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:40449)
0808 04:45:44.316841 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:44.316854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:44.316993 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:44.317307 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:44.317311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:44.317517 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:44.317618 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:44.317624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.318381 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.318385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:44.318963 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:44.318972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.319466 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.319470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:44.319560 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:44.320120 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:44.320133 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:44.322085 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
W20260808 04:45:44.322341 19804 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:44.316603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:40449 (local address 127.4.217.62:33229)
0808 04:45:44.316763 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:44.316768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:44.316928 (+ 160us) server_negotiation.cc:407] Connection header received
0808 04:45:44.317124 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:44.317127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:44.317185 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:44.317278 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:44.317736 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.318263 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.319115 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.319346 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.319458 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:44.322227 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:40449: BlockingRecv error: recv got EOF from 127.4.217.3:40449 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 04:45:44 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:44 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:44.947469 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:44.941910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:58745)
0808 04:45:44.942217 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:44.942234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:44.942356 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:44.942682 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:44.942686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:44.942903 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:44.943001 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:44.943007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.943787 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.943791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:44.944350 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:44.944358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.944873 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.944877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:44.944965 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:44.945484 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:44.945497 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:44.947322 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":128,"threads_started":1}
W20260808 04:45:44.947618 19808 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:44.941977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:58745 (local address 127.4.217.62:33229)
0808 04:45:44.942250 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:44.942255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:44.942291 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:45:44.942478 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:44.942484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:44.942543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:44.942652 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:44.943129 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.943670 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.944474 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:44.944715 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:44.944854 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:44.947459 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:58745: BlockingRecv error: recv got EOF from 127.4.217.1:58745 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
Aug 08 04:45:45 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:45 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:45.141651 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:45.126728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:33601)
0808 04:45:45.127035 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:45.127049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:45.127171 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:45.127495 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:45.127499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:45.127711 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:45.127814 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:45.127819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.128656 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.128659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:45.129267 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:45.129276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.129732 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.129735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:45.129801 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:45.139472 (+ 9671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:45.139485 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:45.141449 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
W20260808 04:45:45.141671 19808 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:45.126788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:33601 (local address 127.4.217.62:33229)
0808 04:45:45.126940 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:45.126944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:45.127109 (+ 165us) server_negotiation.cc:407] Connection header received
0808 04:45:45.127287 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:45.127290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:45.127348 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:45.127439 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:45.127969 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.128528 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.129379 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.129618 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.129703 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:45.141577 (+ 11874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:33601: BlockingRecv error: recv got EOF from 127.4.217.2:33601 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 04:45:45 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:45 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:45.328404 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:45.323342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:38663)
0808 04:45:45.323604 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:45.323619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:45.323727 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:45.323932 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:45.323934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:45.324119 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:45.324218 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:45.324224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.325009 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.325011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:45.325567 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:45.325575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.325971 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.325974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:45.326036 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:45.326517 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:45.326529 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:45.328287 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
W20260808 04:45:45.328545 19808 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:45.323494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:38663 (local address 127.4.217.62:33229)
0808 04:45:45.323626 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:45.323630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:45.323693 (+ 63us) server_negotiation.cc:407] Connection header received
0808 04:45:45.323807 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:45.323809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:45.323854 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:45.323921 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:45.324312 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.324908 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.325668 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.325875 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.325950 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:45.328416 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:38663: BlockingRecv error: recv got EOF from 127.4.217.3:38663 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 04:45:45 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:45 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:45.954133 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:45.948349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:38031)
0808 04:45:45.948728 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:45.948746 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:45.948902 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:45.949220 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:45.949224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:45.949414 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:45.949519 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:45.949525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.950259 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.950262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:45.950834 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:45.950842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.951262 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.951265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:45.951325 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:45.951861 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:45.951887 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:45.953978 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":104,"threads_started":1}
W20260808 04:45:45.954255 19816 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:45.948421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:38031 (local address 127.4.217.62:33229)
0808 04:45:45.948715 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:45.948722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:45.948838 (+ 116us) server_negotiation.cc:407] Connection header received
0808 04:45:45.949017 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:45.949020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:45.949086 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:45.949174 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:45.949624 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.950159 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.950943 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:45.951164 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:45.951239 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:45.954122 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:38031: BlockingRecv error: recv got EOF from 127.4.217.1:38031 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
Aug 08 04:45:46 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:46 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:46.148159 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:46.142501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:35373)
0808 04:45:46.142818 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:46.142831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:46.142936 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:46.143180 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:46.143182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:46.143389 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:46.143490 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:46.143496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.144312 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.144315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:46.145169 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:46.145179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.145592 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.145594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:46.145653 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:46.146151 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:46.146163 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:46.148016 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":122,"threads_started":1}
W20260808 04:45:46.148294 19816 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:46.142555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:35373 (local address 127.4.217.62:33229)
0808 04:45:46.142757 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:46.142764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:46.142910 (+ 146us) server_negotiation.cc:407] Connection header received
0808 04:45:46.143021 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:46.143023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:46.143070 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:46.143139 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:46.143617 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.144210 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.145276 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.145492 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.145588 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:46.148152 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:35373: BlockingRecv error: recv got EOF from 127.4.217.2:35373 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 08 04:45:46 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:46 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:46.334935 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:46.329386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:60577)
0808 04:45:46.329689 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:46.329709 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:46.329812 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:46.330077 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:46.330079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:46.330268 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:46.330368 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:46.330373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.331173 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.331176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:46.331753 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:46.331762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.332240 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.332243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:46.332314 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:46.332936 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:46.332948 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:46.334805 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
W20260808 04:45:46.335172 19816 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:46.329507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:60577 (local address 127.4.217.62:33229)
0808 04:45:46.329656 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:46.329660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:46.329764 (+ 104us) server_negotiation.cc:407] Connection header received
0808 04:45:46.329897 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:46.329899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:46.329964 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:46.330028 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:46.330485 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.331035 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.331867 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.332109 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.332213 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:46.335046 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:60577: BlockingRecv error: recv got EOF from 127.4.217.3:60577 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 04:45:46 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:46 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:46.960623 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:46.954979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:48923)
0808 04:45:46.955290 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:46.955306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:46.955458 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:46.955761 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:46.955765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:46.955996 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:46.956140 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:46.956146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.956993 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.956998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:46.957566 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:46.957574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.958033 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.958036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:46.958119 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:46.958588 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:46.958599 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:46.960431 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":100,"threads_started":1}
W20260808 04:45:46.960680 19820 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:46.955022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:48923 (local address 127.4.217.62:33229)
0808 04:45:46.955290 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:46.955296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:46.955391 (+ 95us) server_negotiation.cc:407] Connection header received
0808 04:45:46.955570 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:46.955575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:46.955626 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:46.955720 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:46.956260 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.956877 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.957685 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:46.957919 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:46.958029 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:46.960559 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:48923: BlockingRecv error: recv got EOF from 127.4.217.1:48923 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
Aug 08 04:45:47 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:47 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:47.155021 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:47.149047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:51377)
0808 04:45:47.149426 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:47.149445 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:47.149568 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:47.149840 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:47.149843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:47.150097 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:47.150227 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:47.150238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.151217 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.151219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:47.151778 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:47.151785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.152215 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.152217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:47.152281 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:47.152768 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:47.152806 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:47.154841 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":105,"threads_started":1}
W20260808 04:45:47.155059 19820 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:47.149121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:51377 (local address 127.4.217.62:33229)
0808 04:45:47.149291 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:47.149298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:47.149567 (+ 269us) server_negotiation.cc:407] Connection header received
0808 04:45:47.149653 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:47.149656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:47.149719 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:47.149808 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:47.150374 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.151100 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.151888 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.152113 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.152193 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:47.154960 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:51377: BlockingRecv error: recv got EOF from 127.4.217.2:51377 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 08 04:45:47 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:47 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:47.341403 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:47.335720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:48477)
0808 04:45:47.335983 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:47.335996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:47.336089 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:47.336318 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:47.336320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:47.336499 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:47.336598 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:47.336604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.337428 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.337431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:47.337999 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:47.338008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.338393 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.338395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:47.338455 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:47.338995 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:47.339007 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:47.341234 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
W20260808 04:45:47.341535 19820 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:47.335768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:48477 (local address 127.4.217.62:33229)
0808 04:45:47.335914 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:47.335918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:47.336060 (+ 142us) server_negotiation.cc:407] Connection header received
0808 04:45:47.336168 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:47.336170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:47.336215 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:47.336283 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:47.336696 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.337329 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.338099 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.338297 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.338368 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:47.341389 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:48477: BlockingRecv error: recv got EOF from 127.4.217.3:48477 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 04:45:47 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:47 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:47.967523 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:47.961639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:39881)
0808 04:45:47.962049 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:47.962071 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:47.962169 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:47.962515 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:47.962517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:47.962707 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:47.962803 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:47.962809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.963929 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.963932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:47.964580 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:47.964591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.965073 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.965076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:47.965136 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:47.965631 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:47.965643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:47.967403 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":105,"threads_started":1}
W20260808 04:45:47.967765 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:47.961787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:39881 (local address 127.4.217.62:33229)
0808 04:45:47.962050 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:47.962054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:47.962141 (+ 87us) server_negotiation.cc:407] Connection header received
0808 04:45:47.962320 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:47.962323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:47.962382 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:47.962477 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:47.962964 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.963763 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.964702 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:47.964975 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:47.965060 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:47.967624 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:39881: BlockingRecv error: recv got EOF from 127.4.217.1:39881 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
Aug 08 04:45:48 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:48 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:48.161507 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:48.155844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:41639)
0808 04:45:48.156143 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:48.156156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:48.156254 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:48.156530 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:48.156532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:48.156725 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:48.156861 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:48.156868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.157680 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.157683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:48.158262 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:48.158269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.158696 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.158699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:48.158759 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:48.159280 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:48.159292 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:48.161381 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
W20260808 04:45:48.161770 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:48.155968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:41639 (local address 127.4.217.62:33229)
0808 04:45:48.156124 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:48.156128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:48.156208 (+ 80us) server_negotiation.cc:407] Connection header received
0808 04:45:48.156323 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:48.156325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:48.156416 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:48.156490 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:48.157038 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.157576 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.158369 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.158595 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.158677 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:48.161627 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:41639: BlockingRecv error: recv got EOF from 127.4.217.2:41639 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 04:45:48 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:48 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:48.348384 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:48.342246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:55443)
0808 04:45:48.342678 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:48.342691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:48.342822 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:48.343133 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:48.343137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:48.343366 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:48.343471 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:48.343477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.344253 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.344258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:48.344858 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:48.344868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.345345 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.345348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:48.345456 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:48.346015 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:48.346034 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:48.348244 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":120,"threads_started":1}
W20260808 04:45:48.348466 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:48.342323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:55443 (local address 127.4.217.62:33229)
0808 04:45:48.342602 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:48.342607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:48.342772 (+ 165us) server_negotiation.cc:407] Connection header received
0808 04:45:48.342963 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:48.342965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:48.343013 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:48.343100 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:48.343599 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.344136 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.344992 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.345221 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.345329 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:48.348373 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:55443: BlockingRecv error: recv got EOF from 127.4.217.3:55443 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 04:45:48 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:48 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:48.974408 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:48.968446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:36811)
0808 04:45:48.968751 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:48.968766 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:48.968939 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:48.969240 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:48.969242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:48.969428 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:48.969528 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:48.969534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.970563 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.970569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:48.971327 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:48.971337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.971748 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.971751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:48.971809 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:48.972302 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:48.972314 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:48.974219 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
W20260808 04:45:48.974452 19830 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:48.968500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:36811 (local address 127.4.217.62:33229)
0808 04:45:48.968826 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:48.968830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:48.968900 (+ 70us) server_negotiation.cc:407] Connection header received
0808 04:45:48.969040 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:48.969045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:48.969104 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:48.969199 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:48.969637 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.970434 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.971435 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:48.971654 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:48.971734 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:48.974343 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:36811: BlockingRecv error: recv got EOF from 127.4.217.1:36811 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":119,"threads_started":1}
Aug 08 04:45:49 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:49 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:49.168596 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:49.162566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:58661)
0808 04:45:49.162895 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:49.162908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:49.163050 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:49.163335 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:49.163338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:49.163583 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:49.163684 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:49.163690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.164470 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.164475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:49.165113 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:49.165123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.165640 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.165645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:49.165782 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:49.166399 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:49.166413 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:49.168406 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":110,"threads_started":1}
W20260808 04:45:49.168718 19830 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:49.162619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:58661 (local address 127.4.217.62:33229)
0808 04:45:49.162812 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:49.162818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:49.162986 (+ 168us) server_negotiation.cc:407] Connection header received
0808 04:45:49.163157 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:49.163160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:49.163216 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:49.163305 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:49.163813 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.164353 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.165242 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.165497 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.165608 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:49.168589 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:58661: BlockingRecv error: recv got EOF from 127.4.217.2:58661 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 08 04:45:49 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:49 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:49.355307 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:49.349275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:49529)
0808 04:45:49.349579 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:49.349592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:49.349710 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:49.349995 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:49.349998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:49.350263 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:49.350386 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:49.350393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.351377 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.351380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:49.351960 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:49.351969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.352420 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.352423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:49.352496 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:49.353082 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:49.353096 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:49.355189 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":116,"threads_started":1}
W20260808 04:45:49.355495 19830 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:49.349342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:49529 (local address 127.4.217.62:33229)
0808 04:45:49.349488 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:49.349493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:49.349672 (+ 179us) server_negotiation.cc:407] Connection header received
0808 04:45:49.349821 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:49.349823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:49.349871 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:49.349938 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:49.350492 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.351262 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.352076 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.352289 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.352371 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:49.355363 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:49529: BlockingRecv error: recv got EOF from 127.4.217.3:49529 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 04:45:49 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:49 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:49.980831 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:49.975270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:55347)
0808 04:45:49.975638 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:49.975652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:49.975753 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:49.976035 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:49.976037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:49.976227 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:49.976321 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:49.976326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.977137 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.977139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:49.977716 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:49.977724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.978179 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.978181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:49.978241 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:49.978795 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:49.978809 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:49.980666 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":100,"threads_started":1}
W20260808 04:45:49.981076 19836 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:49.975344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:55347 (local address 127.4.217.62:33229)
0808 04:45:49.975651 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:49.975655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:49.975730 (+ 75us) server_negotiation.cc:407] Connection header received
0808 04:45:49.975862 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:49.975865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:49.975916 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:49.976003 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:49.976436 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.977040 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.977848 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:49.978074 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:49.978163 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:49.980906 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:55347: BlockingRecv error: recv got EOF from 127.4.217.1:55347 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
Aug 08 04:45:50 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:50 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:50.179904 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:50.169488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:56329)
0808 04:45:50.169867 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:50.169881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:50.170020 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:50.170370 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:50.170374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:50.170603 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:50.170707 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:50.170714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.171552 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.171559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:50.172126 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:50.172135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.172615 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.172619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:50.172707 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:50.177809 (+ 5102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:50.177823 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:50.179782 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":109,"threads_started":1}
W20260808 04:45:50.180083 19836 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:50.169557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:56329 (local address 127.4.217.62:33229)
0808 04:45:50.169743 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:50.169749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:50.169961 (+ 212us) server_negotiation.cc:407] Connection header received
0808 04:45:50.170159 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:50.170162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:50.170214 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:50.170334 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:50.170842 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.171430 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.172254 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.172494 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.172608 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:50.179926 (+ 7318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:56329: BlockingRecv error: recv got EOF from 127.4.217.2:56329 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 04:45:50 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:50 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:50.362028 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:50.356150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:53331)
0808 04:45:50.356504 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:50.356517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:50.356647 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:50.356971 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:50.356974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:50.357193 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:50.357289 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:50.357294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.358153 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.358159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:50.358829 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:50.358839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.359314 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.359318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:50.359407 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:50.359947 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:50.359959 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:50.361913 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":98,"threads_started":1}
W20260808 04:45:50.362286 19836 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:50.356231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:53331 (local address 127.4.217.62:33229)
0808 04:45:50.356400 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:50.356405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:50.356596 (+ 191us) server_negotiation.cc:407] Connection header received
0808 04:45:50.356772 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:50.356796 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:50.356843 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:50.356942 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:50.357419 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.358045 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.358961 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.359197 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.359309 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:50.362115 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:53331: BlockingRecv error: recv got EOF from 127.4.217.3:53331 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 04:45:50 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:50 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:50.988591 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:50.981768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:34231)
0808 04:45:50.982108 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:50.982125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:50.982282 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:50.982572 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:50.982578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:50.982808 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:50.982939 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:50.982950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.983826 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.983832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:50.984938 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:50.984953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.985426 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.985432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:50.985551 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:50.986097 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:50.986117 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:50.988367 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":76,"threads_started":1}
W20260808 04:45:50.988651 19841 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:50.982111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:34231 (local address 127.4.217.62:33229)
0808 04:45:50.982336 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:50.982340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:50.982353 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:50.982392 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:50.982394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:50.982437 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:50.982515 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:50.983094 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.983711 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.985072 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:50.985300 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:50.985384 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:50.988517 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:34231: BlockingRecv error: recv got EOF from 127.4.217.1:34231 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
Aug 08 04:45:51 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:51 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:51.186228 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:51.180663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:57883)
0808 04:45:51.180997 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:51.181011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:51.181112 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:51.181355 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:51.181357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:51.181542 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:51.181642 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:51.181647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.182533 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.182537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:51.183132 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:51.183148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.183576 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.183580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:51.183653 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:51.184160 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:51.184172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:51.186109 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":86,"threads_started":1}
W20260808 04:45:51.186391 19841 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:51.180722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:57883 (local address 127.4.217.62:33229)
0808 04:45:51.180914 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:51.180918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:51.181077 (+ 159us) server_negotiation.cc:407] Connection header received
0808 04:45:51.181192 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:51.181195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:51.181240 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:51.181308 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:51.181747 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.182419 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.183245 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.183451 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.183528 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:51.186238 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:57883: BlockingRecv error: recv got EOF from 127.4.217.2:57883 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 08 04:45:51 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:51 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:51.368583 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:51.362951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:38237)
0808 04:45:51.363302 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:51.363318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:51.363444 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:51.363775 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:51.363778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:51.364045 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:51.364176 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:51.364191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.364987 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.364990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:51.365546 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:51.365553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.365968 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.365971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:51.366036 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:51.366603 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:51.366618 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:51.368450 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":90,"threads_started":1}
W20260808 04:45:51.368762 19841 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:51.363012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:38237 (local address 127.4.217.62:33229)
0808 04:45:51.363184 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:51.363189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:51.363398 (+ 209us) server_negotiation.cc:407] Connection header received
0808 04:45:51.363573 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:51.363577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:51.363640 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:51.363719 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:51.364304 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.364873 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.365644 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.365857 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.365938 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:51.368599 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:38237: BlockingRecv error: recv got EOF from 127.4.217.3:38237 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 04:45:51 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:51 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:51.994899 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:51.989441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:55037)
0808 04:45:51.989757 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:51.989775 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:51.989883 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:51.990125 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:51.990128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:51.990316 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:51.990416 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:51.990421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.991181 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.991184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:51.991753 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:51.991761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.992222 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.992225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:51.992287 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:51.992896 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:51.992909 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:51.994778 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
W20260808 04:45:51.995108 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:51.989499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:55037 (local address 127.4.217.62:33229)
0808 04:45:51.989851 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:51.989856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:51.989872 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:45:51.989967 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:51.989969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:51.990015 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:51.990083 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:51.990522 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.991068 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.991889 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:51.992117 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:51.992201 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:51.994925 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:55037: BlockingRecv error: recv got EOF from 127.4.217.1:55037 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":209,"threads_started":1}
Aug 08 04:45:52 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:52 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:52.193559 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:52.187180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:49149)
0808 04:45:52.187492 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:52.187508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:52.187611 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:52.187908 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:52.187911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:52.188159 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:52.188277 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:52.188284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.189129 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.189132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:52.189862 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:52.189873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.190466 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.190469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:52.190537 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:52.191134 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:52.191146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:52.193378 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":130,"threads_started":1}
W20260808 04:45:52.193647 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:52.187241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:49149 (local address 127.4.217.62:33229)
0808 04:45:52.187406 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:52.187411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:52.187587 (+ 176us) server_negotiation.cc:407] Connection header received
0808 04:45:52.187695 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:52.187697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:52.187746 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:52.187817 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:52.188421 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.188990 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.190014 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.190341 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.190446 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:52.193527 (+ 3081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:49149: BlockingRecv error: recv got EOF from 127.4.217.2:49149 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 04:45:52 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:52 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:52.375145 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:52.369353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:52985)
0808 04:45:52.369703 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:52.369717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:52.369849 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:52.370134 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:52.370137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:52.370344 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:52.370438 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:52.370443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.371209 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.371215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:52.371768 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:52.371775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.372244 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.372248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:52.372334 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:52.372842 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:52.372854 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:52.374968 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
W20260808 04:45:52.375180 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:52.369396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:52985 (local address 127.4.217.62:33229)
0808 04:45:52.369568 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:52.369574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:52.369794 (+ 220us) server_negotiation.cc:407] Connection header received
0808 04:45:52.369960 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:52.369964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:52.370021 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:52.370107 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:52.370565 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.371097 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.371888 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.372125 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.372239 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:52.375077 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:52985: BlockingRecv error: recv got EOF from 127.4.217.3:52985 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 04:45:52 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:52 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:53.001266 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:52.995706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:58367)
0808 04:45:52.995991 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:52.996004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:52.996099 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:52.996356 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:52.996358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:52.996538 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:52.996639 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:52.996644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.997440 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.997443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:52.997990 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:52.997998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.998462 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.998464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:52.998525 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:52.999081 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:52.999094 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:53.001087 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
W20260808 04:45:53.001430 19851 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:52.995766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:58367 (local address 127.4.217.62:33229)
0808 04:45:52.996033 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:52.996036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:52.996049 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:52.996195 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:52.996198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:52.996243 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:52.996320 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:52.996745 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.997341 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.998113 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:52.998357 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:52.998448 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:53.001254 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:58367: BlockingRecv error: recv got EOF from 127.4.217.1:58367 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
Aug 08 04:45:53 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:53 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:53.200314 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:53.194685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:47449)
0808 04:45:53.195007 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:53.195022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:53.195107 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:53.195331 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:53.195333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:53.195534 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:53.195632 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:53.195637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:53.196333 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:53.196336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:53.197008 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:53.197021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:53.197542 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:53.197546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:53.197627 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:53.198195 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:53.198210 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:53.200090 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":73,"threads_started":1}
W20260808 04:45:53.200352 19851 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:53.194915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:47449 (local address 127.4.217.62:33229)
0808 04:45:53.195073 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:53.195077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:53.195090 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:53.195177 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:53.195180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:53.195223 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:53.195288 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:53.195733 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:53.196243 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:53.197146 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:53.197417 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:53.197536 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:53.200218 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:47449: BlockingRecv error: recv got EOF from 127.4.217.2:47449 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 04:45:53 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:53 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:53.381500 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:53.375961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:38727)
0808 04:45:53.376257 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:53.376272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:53.376369 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:53.376618 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:53.376620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:53.376825 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:53.376927 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:53.376932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:53.377649 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:53.377651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:53.378194 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:53.378201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:53.378615 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:53.378617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:53.378677 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:53.379137 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:53.379149 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:53.381324 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
W20260808 04:45:53.381556 19851 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:53.376067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:38727 (local address 127.4.217.62:33229)
0808 04:45:53.376258 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:53.376262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:53.376343 (+ 81us) server_negotiation.cc:407] Connection header received
0808 04:45:53.376474 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:53.376477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:53.376523 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:53.376590 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:53.377027 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:53.377551 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:53.378293 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:53.378507 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:53.378582 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:53.381446 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:38727: BlockingRecv error: recv got EOF from 127.4.217.3:38727 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 08 04:45:54 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:54 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:54.008239 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:54.002313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:49897)
0808 04:45:54.002642 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:54.002656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:54.002754 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:54.003027 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:54.003030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:54.003225 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:54.003326 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:54.003332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.004150 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.004154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:54.004755 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:54.004767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.005464 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.005470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:54.005564 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:54.006115 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:54.006127 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:54.008053 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
W20260808 04:45:54.008304 19856 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:54.002442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:49897 (local address 127.4.217.62:33229)
0808 04:45:54.002694 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:54.002698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:54.002712 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:45:54.002849 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:54.002851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:54.002907 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:54.002989 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:54.003434 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.004002 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.004920 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.005257 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.005383 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:54.008184 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:49897: BlockingRecv error: recv got EOF from 127.4.217.1:49897 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
Aug 08 04:45:54 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:54 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:54.206705 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:54.201125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:40629)
0808 04:45:54.201412 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:54.201432 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:54.201574 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:54.201853 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:54.201856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:54.202069 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:54.202167 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:54.202173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.202945 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.202950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:54.203551 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:54.203559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.204012 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.204016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:54.204103 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:54.204700 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:54.204712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:54.206587 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
W20260808 04:45:54.206878 19856 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:54.201191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:40629 (local address 127.4.217.62:33229)
0808 04:45:54.201377 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:54.201382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:54.201507 (+ 125us) server_negotiation.cc:407] Connection header received
0808 04:45:54.201679 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:54.201683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:54.201736 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:54.201826 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:54.202290 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.202832 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.203674 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.203896 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.204007 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:54.206722 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:40629: BlockingRecv error: recv got EOF from 127.4.217.2:40629 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 08 04:45:54 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:54 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:54.388056 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:54.382385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:60849)
0808 04:45:54.382687 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:54.382701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:54.382808 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:54.383085 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:54.383088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:54.383276 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:54.383378 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:54.383383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.384103 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.384105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:54.384646 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:54.384653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.385088 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.385091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:54.385156 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:54.385724 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:54.385736 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:54.387854 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
W20260808 04:45:54.388096 19856 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:54.382465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:60849 (local address 127.4.217.62:33229)
0808 04:45:54.382712 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:54.382716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:54.382770 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:45:54.382904 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:54.382906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:54.382957 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:54.383030 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:54.383480 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.384001 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.384746 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:54.384980 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:54.385057 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:54.387984 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:60849: BlockingRecv error: recv got EOF from 127.4.217.3:60849 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Aug 08 04:45:55 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:55 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:55.015105 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:55.009154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:57441)
0808 04:45:55.009455 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:55.009469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:55.009564 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:55.009822 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:55.009825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:55.010041 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:55.010141 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:55.010146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.010872 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.010875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:55.011437 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:55.011444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.011857 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.011859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:55.011932 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:55.012604 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:55.012616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:55.014920 (+ 2304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":92,"threads_started":1}
W20260808 04:45:55.015164 19860 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:55.009244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:57441 (local address 127.4.217.62:33229)
0808 04:45:55.009491 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:55.009494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:55.009511 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:45:55.009660 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:55.009663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:55.009709 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:55.009788 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:55.010243 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.010781 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.011537 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.011759 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.011837 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:55.015039 (+ 3202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:57441: BlockingRecv error: recv got EOF from 127.4.217.1:57441 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
Aug 08 04:45:55 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:55 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:55.216251 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:55.207441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:34941)
0808 04:45:55.207787 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:55.207800 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:55.207957 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:55.208234 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:55.208237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:55.208472 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:55.208586 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:55.208592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.209450 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.209456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:55.210029 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:55.210038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.210520 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.210523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:55.210615 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:55.214082 (+ 3467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:55.214099 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:55.216078 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":117,"threads_started":1}
W20260808 04:45:55.216308 19860 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:55.207559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:34941 (local address 127.4.217.62:33229)
0808 04:45:55.207739 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:55.207744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:55.207886 (+ 142us) server_negotiation.cc:407] Connection header received
0808 04:45:55.208063 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:55.208066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:55.208121 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:55.208210 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:55.208713 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.209323 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.210167 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.210400 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.210505 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:55.216201 (+ 5696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:34941: BlockingRecv error: recv got EOF from 127.4.217.2:34941 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 08 04:45:55 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:55 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:55.394879 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:55.389091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:54957)
0808 04:45:55.389401 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:55.389415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:55.389524 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:55.389805 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:55.389807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:55.389994 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:55.390118 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:55.390123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.390851 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.390853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:55.391411 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:55.391419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.391827 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.391829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:55.391890 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:55.392503 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:55.392515 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:55.394740 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
W20260808 04:45:55.395035 19860 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:55.389190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:54957 (local address 127.4.217.62:33229)
0808 04:45:55.389392 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:55.389397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:55.389490 (+ 93us) server_negotiation.cc:407] Connection header received
0808 04:45:55.389631 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:55.389635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:55.389692 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:55.389762 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:55.390220 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.390735 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.391517 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:55.391728 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:55.391806 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:55.394914 (+ 3108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:54957: BlockingRecv error: recv got EOF from 127.4.217.3:54957 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 08 04:45:56 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:56 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:56.021644 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:56.015932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:59479)
0808 04:45:56.016231 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:56.016247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:56.016355 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:56.016608 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:56.016611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:56.016824 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:56.016956 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:56.016962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.017694 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.017696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:56.018285 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:56.018294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.018701 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.018704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:56.018762 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:56.019350 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:56.019362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:56.021482 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
W20260808 04:45:56.021785 19865 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:56.016076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:59479 (local address 127.4.217.62:33229)
0808 04:45:56.016328 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:56.016332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:56.016345 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:45:56.016443 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:56.016446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:56.016490 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:56.016571 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:56.017067 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.017599 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.018391 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.018605 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.018680 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:56.021637 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:59479: BlockingRecv error: recv got EOF from 127.4.217.1:59479 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
Aug 08 04:45:56 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:56 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:56.223351 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:56.217066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:32861)
0808 04:45:56.217409 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:56.217428 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:56.217597 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:56.217899 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:56.217903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:56.218143 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:56.218252 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:56.218259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.219104 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.219110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:56.219802 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:56.219821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.220510 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.220517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:56.220609 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:56.221270 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:56.221284 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:56.223228 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
W20260808 04:45:56.223551 19865 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:56.217238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:32861 (local address 127.4.217.62:33229)
0808 04:45:56.217411 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:56.217417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:56.217526 (+ 109us) server_negotiation.cc:407] Connection header received
0808 04:45:56.217709 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:56.217714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:56.217774 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:56.217871 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:56.218395 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.218973 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.219939 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.220247 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.220389 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:56.223403 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:32861: BlockingRecv error: recv got EOF from 127.4.217.2:32861 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 04:45:56 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:56 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:56.401738 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:56.395817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:57311)
0808 04:45:56.396135 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:56.396149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:56.396254 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:56.396494 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:56.396497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:56.396688 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:56.396812 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:56.396818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.397601 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.397605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:56.398191 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:56.398200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.398670 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.398673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:56.398759 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:56.399333 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:56.399345 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:56.401550 (+ 2205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
W20260808 04:45:56.401799 19865 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:56.395867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:57311 (local address 127.4.217.62:33229)
0808 04:45:56.396023 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:56.396028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:56.396224 (+ 196us) server_negotiation.cc:407] Connection header received
0808 04:45:56.396338 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:56.396340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:56.396390 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:56.396459 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:56.396933 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.397461 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.398318 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:56.398550 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:56.398664 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:56.401680 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:57311: BlockingRecv error: recv got EOF from 127.4.217.3:57311 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 04:45:57 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:57 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:57.028020 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:57.022518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:33087)
0808 04:45:57.022843 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:57.022860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:57.022973 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:57.023234 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:57.023237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:57.023437 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:57.023538 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:57.023544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.024330 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.024333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:57.024982 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:57.024993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.025419 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.025422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:57.025483 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:57.025961 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:57.025974 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:57.027889 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
W20260808 04:45:57.028206 19871 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:57.022567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:33087 (local address 127.4.217.62:33229)
0808 04:45:57.022908 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:57.022911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:57.022925 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:45:57.023068 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:57.023071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:57.023115 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:57.023183 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:57.023647 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.024204 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.025098 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.025318 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.025400 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:57.028023 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:33087: BlockingRecv error: recv got EOF from 127.4.217.1:33087 (error 108)
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
Aug 08 04:45:57 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:57 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:57.230155 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:57.224296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:49557)
0808 04:45:57.224847 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:57.224863 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:57.224972 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:57.225319 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:57.225323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:57.225576 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:57.225710 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:57.225717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.226563 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.226566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:57.227140 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:57.227148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.227597 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.227600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:57.227663 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:57.228207 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:57.228222 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:57.230019 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":112,"threads_started":1}
W20260808 04:45:57.230312 19871 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:57.224386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:49557 (local address 127.4.217.62:33229)
0808 04:45:57.224582 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:57.224586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:57.224947 (+ 361us) server_negotiation.cc:407] Connection header received
0808 04:45:57.225113 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:57.225117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:57.225179 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:57.225259 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:57.225880 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.226457 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.227259 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.227492 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.227576 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:57.230179 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:49557: BlockingRecv error: recv got EOF from 127.4.217.2:49557 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 04:45:57 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:57 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.3@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:57.408573 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:57.402527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.3:51801)
0808 04:45:57.402873 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:57.402888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:57.403034 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:57.403339 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:57.403342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:57.403562 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:57.403661 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:57.403666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.404428 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.404433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:57.405113 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:57.405122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.405620 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.405624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:57.405706 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:57.406273 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:57.406295 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:57.408426 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
W20260808 04:45:57.408644 19871 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:57.402655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:51801 (local address 127.4.217.62:33229)
0808 04:45:57.402824 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:57.402829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:57.402980 (+ 151us) server_negotiation.cc:407] Connection header received
0808 04:45:57.403163 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:57.403167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:57.403224 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:57.403312 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:57.403784 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.404314 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.405236 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:57.405460 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:57.405618 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:57.408532 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.3:51801: BlockingRecv error: recv got EOF from 127.4.217.3:51801 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 04:45:58 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:58 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.1@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:58.034917 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:58.028997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.1:36235)
0808 04:45:58.029330 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:58.029345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:58.029435 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:58.029692 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:58.029695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:58.029883 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:58.029985 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:58.029991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.030779 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.030782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:58.031343 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:58.031352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.031825 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.031827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:58.031888 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:58.032421 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:58.032433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:58.034744 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
W20260808 04:45:58.034984 19879 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:58.029110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:36235 (local address 127.4.217.62:33229)
0808 04:45:58.029418 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:58.029421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:58.029437 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:45:58.029519 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:58.029522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:58.029568 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:58.029655 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:58.030117 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.030679 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.031470 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.031712 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.031840 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:58.034876 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.1:36235: BlockingRecv error: recv got EOF from 127.4.217.1:36235 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":85,"threads_started":1}
Aug 08 04:45:58 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:45:58 dist-test-slave-2kcd krb5kdc[18186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.217.2@KRBTEST.COM for oryx/127.4.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:45:58.241297 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:58.231618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33229 (local address 127.4.217.2:37573)
0808 04:45:58.232200 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:58.232238 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:58.232455 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:58.233043 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:58.233046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:58.233537 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:58.233696 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:58.233703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.234975 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.234991 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:58.235932 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:58.235950 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.236732 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.236737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:58.236951 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:58.237847 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:45:58.237873 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:58.240927 (+ 3054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.217.62:33229: Server oryx/127.4.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":154,"threads_started":1}
W20260808 04:45:58.241465 19879 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:58.231812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:37573 (local address 127.4.217.62:33229)
0808 04:45:58.232166 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:58.232173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:58.232349 (+ 176us) server_negotiation.cc:407] Connection header received
0808 04:45:58.232659 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:58.232664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:58.232772 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:58.233021 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:58.233903 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.234740 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.236157 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.236533 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.236724 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:58.241184 (+ 4460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.217.2:37573: BlockingRecv error: recv got EOF from 127.4.217.2:37573 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20260808 04:45:58.309145 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 19136
I20260808 04:45:58.360466 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 19391
I20260808 04:45:58.385095 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 19598
I20260808 04:45:58.401417 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 19033
2026-08-08T04:45:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18398 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T04:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:58Z Disabled control of system clock
I20260808 04:45:58.456091 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:46521
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:46521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 04:45:58.653774 19891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:58.654250 19891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:58.654311 19891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:58.660452 19891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:58.660561 19891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:58.660599 19891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:58.660686 19891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:58.660768 19891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:58.668166 19891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:46521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:46521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:58.670070 19891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:58.672462 19891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:58.687299 19901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:58.687264 19891 server_base.cc:1034] running on GCE node
W20260808 04:45:58.687228 19898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:58.687261 19899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:58.688556 19891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:58.690378 19891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:58.691576 19891 hybrid_clock.cc:648] HybridClock initialized: now 1786164358691546 us; error 49 us; skew 500 ppm
I20260808 04:45:58.693709 19891 webserver.cc:460] Webserver started at http://127.4.217.62:32903/ using document root <none> and password file <none>
I20260808 04:45:58.694265 19891 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:58.694324 19891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:58.694500 19891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:58.696586 19891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "826451652a9e4fc5a6872bf7d56d77a2"
format_stamp: "Formatted at 2026-08-08 04:45:58 on dist-test-slave-2kcd"
server_key: "15437b3f0bc5a0bfe8815cca4abf431a"
server_key_iv: "60a15f9148bb1bed5d7c91d75df629b7"
server_key_version: "encryptionkey@0"
I20260808 04:45:58.697161 19891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "826451652a9e4fc5a6872bf7d56d77a2"
format_stamp: "Formatted at 2026-08-08 04:45:58 on dist-test-slave-2kcd"
server_key: "15437b3f0bc5a0bfe8815cca4abf431a"
server_key_iv: "60a15f9148bb1bed5d7c91d75df629b7"
server_key_version: "encryptionkey@0"
I20260808 04:45:58.700824 19891 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:45:58.703186 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:58.704404 19891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:58.704545 19891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "826451652a9e4fc5a6872bf7d56d77a2"
format_stamp: "Formatted at 2026-08-08 04:45:58 on dist-test-slave-2kcd"
server_key: "15437b3f0bc5a0bfe8815cca4abf431a"
server_key_iv: "60a15f9148bb1bed5d7c91d75df629b7"
server_key_version: "encryptionkey@0"
I20260808 04:45:58.704658 19891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:58.716401 19891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:58.717166 19891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:58.717366 19891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:58.724330 19891 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:46521
I20260808 04:45:58.724337 19992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:46521 every 8 connection(s)
I20260808 04:45:58.725337 19891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:58.726701 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 19891
I20260808 04:45:58.726817 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:58.727164 4964 external_mini_cluster.cc:1428] Setting key 3f69511521ef8a95c2ab76e060956930
I20260808 04:45:58.728834 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:58.734614 19994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2: Bootstrap starting.
I20260808 04:45:58.736356 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:58.728573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.4.217.62:46521)
0808 04:45:58.729290 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:58.729305 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:58.729339 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:45:58.730069 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:58.730090 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:58.730489 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:58.730806 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:58.731228 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.732184 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.733140 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.733605 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.733781 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:58.734662 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:58.734667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:58.734699 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:58.735093 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:58.735147 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:58.735168 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:58.735278 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:58.735444 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 04:45:58.735684 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":169,"threads_started":1}
I20260808 04:45:58.737007 19994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:58.738049 19994 log.cc:824] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:58.739913 19994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2: No bootstrap required, opened a new log
I20260808 04:45:58.743281 19994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826451652a9e4fc5a6872bf7d56d77a2" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46521 } }
I20260808 04:45:58.743480 19994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:58.743568 19994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826451652a9e4fc5a6872bf7d56d77a2, State: Initialized, Role: FOLLOWER
I20260808 04:45:58.744041 19994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826451652a9e4fc5a6872bf7d56d77a2" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46521 } }
I20260808 04:45:58.744166 19994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:58.744249 19994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:58.744367 19994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:58.745304 19994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826451652a9e4fc5a6872bf7d56d77a2" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46521 } }
I20260808 04:45:58.745644 19994 leader_election.cc:302] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 826451652a9e4fc5a6872bf7d56d77a2; no voters:
I20260808 04:45:58.745934 19994 leader_election.cc:288] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:58.746098 20002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:58.746395 20002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [term 1 LEADER]: Becoming Leader. State: Replica: 826451652a9e4fc5a6872bf7d56d77a2, State: Running, Role: LEADER
I20260808 04:45:58.746712 20002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826451652a9e4fc5a6872bf7d56d77a2" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46521 } }
I20260808 04:45:58.747254 19994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:58.747839 20004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 826451652a9e4fc5a6872bf7d56d77a2. Latest consensus state: current_term: 1 leader_uuid: "826451652a9e4fc5a6872bf7d56d77a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826451652a9e4fc5a6872bf7d56d77a2" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46521 } } }
I20260808 04:45:58.748068 20004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:58.748293 20003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "826451652a9e4fc5a6872bf7d56d77a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826451652a9e4fc5a6872bf7d56d77a2" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46521 } } }
I20260808 04:45:58.748397 20003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:58.748648 20013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:58.751541 20013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:58.757035 20013 catalog_manager.cc:1347] Generated new cluster ID: dfdb50ecfd7440d4b0c16a805af91de2
I20260808 04:45:58.757109 20013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:58.767114 20013 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:58.768217 20013 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:58.777042 20013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 826451652a9e4fc5a6872bf7d56d77a2: Generated new TSK 0
I20260808 04:45:58.777691 20013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:45:58.786182 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46521
--builtin_ntp_servers=127.4.217.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 04:45:58.903510 20032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:58.903740 20032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:58.903774 20032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:58.907761 20032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:58.907836 20032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:58.907898 20032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:58.912737 20032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:58.913707 20032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:58.915002 20032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:45:58.924760 20032 server_base.cc:1034] running on GCE node
W20260808 04:45:58.924702 20042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:58.924686 20040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:58.924686 20046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:58.925468 20032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:58.926108 20032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:58.927299 20032 hybrid_clock.cc:648] HybridClock initialized: now 1786164358927285 us; error 25 us; skew 500 ppm
I20260808 04:45:58.929178 20032 webserver.cc:460] Webserver started at http://127.4.217.1:42827/ using document root <none> and password file <none>
I20260808 04:45:58.929792 20032 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:58.929885 20032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:58.930128 20032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:58.932173 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "ccf35fd0174a43c38fc22075e5d1e50a"
format_stamp: "Formatted at 2026-08-08 04:45:58 on dist-test-slave-2kcd"
server_key: "f82b2f6065a041bf95ebe9f7fa3e0ea7"
server_key_iv: "10a7d7af24050fae080f9bfd73948b85"
server_key_version: "encryptionkey@0"
I20260808 04:45:58.932749 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "ccf35fd0174a43c38fc22075e5d1e50a"
format_stamp: "Formatted at 2026-08-08 04:45:58 on dist-test-slave-2kcd"
server_key: "f82b2f6065a041bf95ebe9f7fa3e0ea7"
server_key_iv: "10a7d7af24050fae080f9bfd73948b85"
server_key_version: "encryptionkey@0"
I20260808 04:45:58.936548 20032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:45:58.939113 20054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:58.940364 20032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:58.940513 20032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "ccf35fd0174a43c38fc22075e5d1e50a"
format_stamp: "Formatted at 2026-08-08 04:45:58 on dist-test-slave-2kcd"
server_key: "f82b2f6065a041bf95ebe9f7fa3e0ea7"
server_key_iv: "10a7d7af24050fae080f9bfd73948b85"
server_key_version: "encryptionkey@0"
I20260808 04:45:58.940630 20032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:58.971611 20032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:58.972374 20032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:58.972596 20032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:58.973302 20032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:58.974299 20032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:58.974372 20032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:58.974462 20032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:58.974519 20032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:58.983918 20032 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:35411
I20260808 04:45:58.983971 20240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:35411 every 8 connection(s)
I20260808 04:45:58.984897 20032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:58.993975 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:58.986799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:33463 (local address 127.4.217.62:46521)
0808 04:45:58.986960 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:58.986965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:58.988080 (+ 1115us) server_negotiation.cc:407] Connection header received
0808 04:45:58.989619 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:58.989624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:58.989688 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:58.989765 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:58.990646 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.991166 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.992260 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.992473 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.992551 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:58.993553 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:58.993556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:58.993558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:58.993570 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:58.993589 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:58.993597 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:58.993654 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:58.993794 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 04:45:58.993858 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:45:58.994159 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 20032
I20260808 04:45:58.994288 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:58.994637 4964 external_mini_cluster.cc:1428] Setting key d201054a4f8a6b95bfc1c3ddd014248d
I20260808 04:45:58.995061 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:58.987253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:46521 (local address 127.4.217.1:33463)
0808 04:45:58.987922 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:58.987965 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:58.989365 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:58.989934 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:58.989940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:58.990020 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:58.990518 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:58.990530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.991278 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.991281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:58.992154 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:58.992164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:58.992607 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:58.992610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:58.992757 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:58.993314 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:58.993315 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:58.993354 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:58.993415 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:58.993671 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:58.993678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:58.993682 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:45:58.993777 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 04:45:58.994078 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260808 04:45:58.996368 20241 heartbeater.cc:347] Connected to a master server at 127.4.217.62:46521
I20260808 04:45:58.996608 20241 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:58.996869 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46521
--builtin_ntp_servers=127.4.217.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 04:45:58.997257 20241 heartbeater.cc:511] Master 127.4.217.62:46521 requested a full tablet report, sending...
I20260808 04:45:58.998827 19939 ts_manager.cc:194] Registered new tserver with Master: ccf35fd0174a43c38fc22075e5d1e50a (127.4.217.1:35411)
I20260808 04:45:59.000000 19939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.1:33463
W20260808 04:45:59.114732 20247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:59.115010 20247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:59.115048 20247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:59.119056 20247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:59.119112 20247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:59.119163 20247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:45:59.123965 20247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:59.125056 20247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:59.126410 20247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:59.136032 20256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:59.136113 20258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:59.136133 20255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:59.136189 20247 server_base.cc:1034] running on GCE node
I20260808 04:45:59.136760 20247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:59.137457 20247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:59.138648 20247 hybrid_clock.cc:648] HybridClock initialized: now 1786164359138617 us; error 51 us; skew 500 ppm
I20260808 04:45:59.140476 20247 webserver.cc:460] Webserver started at http://127.4.217.2:44683/ using document root <none> and password file <none>
I20260808 04:45:59.141153 20247 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:59.141258 20247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:59.141505 20247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:59.143755 20247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "4fa1ad1995634de19cca750caf846839"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "81b54362fb82b691d1468e90ca5e51d4"
server_key_iv: "daceb94976cd232a2a2ea141d022f3a0"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.144363 20247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "4fa1ad1995634de19cca750caf846839"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "81b54362fb82b691d1468e90ca5e51d4"
server_key_iv: "daceb94976cd232a2a2ea141d022f3a0"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.148432 20247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260808 04:45:59.151083 20269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.152269 20247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:59.152410 20247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "4fa1ad1995634de19cca750caf846839"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "81b54362fb82b691d1468e90ca5e51d4"
server_key_iv: "daceb94976cd232a2a2ea141d022f3a0"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.152525 20247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:59.187390 20247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:59.188194 20247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:59.188419 20247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:59.189116 20247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:59.190137 20247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:59.190213 20247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.190284 20247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:59.190340 20247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.199515 20247 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:32913
I20260808 04:45:59.199563 20440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:32913 every 8 connection(s)
I20260808 04:45:59.200452 20247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:45:59.205276 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 20247
I20260808 04:45:59.205399 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:45:59.205734 4964 external_mini_cluster.cc:1428] Setting key ab9f6948d1a89cbbfb6ca4bae0747bfe
I20260808 04:45:59.208087 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46521
--builtin_ntp_servers=127.4.217.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 04:45:59.210847 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:59.202383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:53753 (local address 127.4.217.62:46521)
0808 04:45:59.202535 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:59.202539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:59.203605 (+ 1066us) server_negotiation.cc:407] Connection header received
0808 04:45:59.205118 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:59.205121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:59.205186 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:59.205276 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:59.206598 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:59.207388 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:59.208733 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:59.209114 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:59.209227 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:59.210337 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:59.210341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:59.210351 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:59.210364 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:59.210391 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:59.210400 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:59.210479 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:59.210644 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 04:45:59.210707 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:45:59.211711 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:59.202789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:46521 (local address 127.4.217.2:53753)
0808 04:45:59.203444 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:59.203485 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:59.204772 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:59.205494 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:59.205503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:59.205626 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:59.206414 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:59.206443 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:59.207554 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:59.207557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:59.208608 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:59.208618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:59.209269 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:59.209273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:59.209438 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:59.210074 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:59.210076 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:59.210114 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:59.210177 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:59.210481 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:59.210494 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:59.210503 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:45:59.210641 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 04:45:59.210938 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260808 04:45:59.212831 20443 heartbeater.cc:347] Connected to a master server at 127.4.217.62:46521
I20260808 04:45:59.213056 20443 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:59.213572 20443 heartbeater.cc:511] Master 127.4.217.62:46521 requested a full tablet report, sending...
I20260808 04:45:59.214633 19939 ts_manager.cc:194] Registered new tserver with Master: 4fa1ad1995634de19cca750caf846839 (127.4.217.2:32913)
I20260808 04:45:59.215201 19939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.2:53753
W20260808 04:45:59.322397 20446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:59.322666 20446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:59.322733 20446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:59.326936 20446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:59.327014 20446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:59.327107 20446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:45:59.331892 20446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:46521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:59.332976 20446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:59.334343 20446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:59.343950 20454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:59.343950 20455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:59.343981 20458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:59.344215 20446 server_base.cc:1034] running on GCE node
I20260808 04:45:59.344712 20446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:59.345372 20446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:59.346534 20446 hybrid_clock.cc:648] HybridClock initialized: now 1786164359346499 us; error 46 us; skew 500 ppm
I20260808 04:45:59.348536 20446 webserver.cc:460] Webserver started at http://127.4.217.3:33141/ using document root <none> and password file <none>
I20260808 04:45:59.349195 20446 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:59.349258 20446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:59.349485 20446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:59.351536 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "2c75ee90ddfd43e5ac701a44b2aa3a71"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "03e0d50f45862b20d45aac6b6a70d81f"
server_key_iv: "f23bbd35fbd7a51a897b6d4513ad5a3f"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.352098 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "2c75ee90ddfd43e5ac701a44b2aa3a71"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "03e0d50f45862b20d45aac6b6a70d81f"
server_key_iv: "f23bbd35fbd7a51a897b6d4513ad5a3f"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.355895 20446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:45:59.358526 20469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.359789 20446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:45:59.359886 20446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "2c75ee90ddfd43e5ac701a44b2aa3a71"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "03e0d50f45862b20d45aac6b6a70d81f"
server_key_iv: "f23bbd35fbd7a51a897b6d4513ad5a3f"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.360044 20446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:59.382598 20446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:59.383312 20446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:59.383514 20446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:59.384120 20446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:59.385097 20446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:59.385145 20446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.385185 20446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:59.385201 20446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.394392 20446 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:43201
I20260808 04:45:59.394434 20646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:43201 every 8 connection(s)
I20260808 04:45:59.395257 20446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:45:59.395893 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 20446
I20260808 04:45:59.396006 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:45:59.396311 4964 external_mini_cluster.cc:1428] Setting key 29caff256fac010afe708641405af235
I20260808 04:45:59.404531 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:59.397257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:41603 (local address 127.4.217.62:46521)
0808 04:45:59.397418 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:59.397423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:59.398703 (+ 1280us) server_negotiation.cc:407] Connection header received
0808 04:45:59.400038 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:59.400041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:59.400087 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:59.400156 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:59.401064 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:59.401592 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:59.402693 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:45:59.402896 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:45:59.402976 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:59.404107 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:59.404111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:59.404113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:59.404125 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:59.404144 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:59.404152 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:59.404221 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:59.404366 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 04:45:59.404418 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:45:59.405370 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:59.397785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:46521 (local address 127.4.217.3:41603)
0808 04:45:59.398542 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:59.398592 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:59.399850 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:59.400286 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:59.400292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:59.400369 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:59.400938 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:59.400955 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:59.401704 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:59.401706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:59.402587 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:45:59.402597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:45:59.403049 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:45:59.403053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:45:59.403217 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:45:59.403753 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:59.403755 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:59.403793 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:59.403854 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:59.404238 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:59.404245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:59.404250 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:45:59.404346 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:45:59.404565 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":168,"threads_started":1}
I20260808 04:45:59.406458 20648 heartbeater.cc:347] Connected to a master server at 127.4.217.62:46521
I20260808 04:45:59.406665 20648 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:59.407152 20648 heartbeater.cc:511] Master 127.4.217.62:46521 requested a full tablet report, sending...
I20260808 04:45:59.408218 19939 ts_manager.cc:194] Registered new tserver with Master: 2c75ee90ddfd43e5ac701a44b2aa3a71 (127.4.217.3:43201)
I20260808 04:45:59.408893 19939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.3:41603
I20260808 04:45:59.410743 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:45:59.413686 19998 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:45:59.412883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.4.217.62:46521)
0808 04:45:59.413006 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:59.413009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:59.413221 (+ 212us) server_negotiation.cc:407] Connection header received
0808 04:45:59.413300 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:59.413302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:59.413349 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:59.413420 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:59.413596 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57828: BlockingRecv error: recv got EOF from 127.0.0.1:57828 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:45:59.413968 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 20032
I20260808 04:45:59.435806 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 20247
I20260808 04:45:59.447782 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 20446
I20260808 04:45:59.463361 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 19891
2026-08-08T04:45:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1048 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T04:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:45:59Z Disabled control of system clock
I20260808 04:45:59.498723 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35873
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:37917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:45:59.610288 20666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:59.610523 20666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:59.610558 20666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:59.614651 20666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:45:59.614710 20666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:59.614727 20666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:59.614743 20666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:45:59.614814 20666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:45:59.619544 20666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:59.620575 20666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:59.621960 20666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:59.630666 20679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:59.630656 20676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:59.630618 20675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:59.631004 20666 server_base.cc:1034] running on GCE node
I20260808 04:45:59.631518 20666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:59.632550 20666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:59.633723 20666 hybrid_clock.cc:648] HybridClock initialized: now 1786164359633695 us; error 39 us; skew 500 ppm
I20260808 04:45:59.635579 20666 webserver.cc:460] Webserver started at http://127.4.217.62:33135/ using document root <none> and password file <none>
I20260808 04:45:59.636104 20666 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:59.636162 20666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:59.636327 20666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:59.638418 20666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "a91b8ebeed3a4f3f8467d506594650d0"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "2204db387b0e6e61a52990cbc8955f81"
server_key_iv: "483d956ce64125c22b8167b3e98f2add"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.638923 20666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "a91b8ebeed3a4f3f8467d506594650d0"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "2204db387b0e6e61a52990cbc8955f81"
server_key_iv: "483d956ce64125c22b8167b3e98f2add"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.642671 20666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:45:59.645107 20690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.646296 20666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:45:59.646453 20666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "a91b8ebeed3a4f3f8467d506594650d0"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "2204db387b0e6e61a52990cbc8955f81"
server_key_iv: "483d956ce64125c22b8167b3e98f2add"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.646570 20666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:59.651221 20666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:59.651819 20666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:59.652004 20666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:59.658977 20666 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:35873
I20260808 04:45:59.659030 20776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:35873 every 8 connection(s)
I20260808 04:45:59.659910 20666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:45:59.663161 20781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:45:59.664336 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 20666
I20260808 04:45:59.664433 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:45:59.664722 4964 external_mini_cluster.cc:1428] Setting key 082ef1125124444b8f03bae1e2bf75ab
I20260808 04:45:59.669317 20781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0: Bootstrap starting.
I20260808 04:45:59.670197 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:59.666202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.4.217.62:35873)
0808 04:45:59.666809 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:59.666825 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:59.666859 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:45:59.667467 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:59.667482 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:59.667810 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:59.668073 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:59.668961 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:59.668966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:59.668977 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:59.668988 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:59.669045 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:59.669061 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:59.669171 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:59.669299 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 04:45:59.669564 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":138,"threads_started":1}
I20260808 04:45:59.671561 20781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0: Neither blocks nor log segments found. Creating new log.
I20260808 04:45:59.672381 20781 log.cc:824] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0: Log is configured to *not* fsync() on all Append() calls
I20260808 04:45:59.674248 20781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0: No bootstrap required, opened a new log
I20260808 04:45:59.677837 20781 raft_consensus.cc:348] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b8ebeed3a4f3f8467d506594650d0" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35873 } }
I20260808 04:45:59.678200 20781 raft_consensus.cc:374] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:45:59.678282 20781 raft_consensus.cc:729] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b8ebeed3a4f3f8467d506594650d0, State: Initialized, Role: FOLLOWER
I20260808 04:45:59.678741 20781 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b8ebeed3a4f3f8467d506594650d0" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35873 } }
I20260808 04:45:59.678893 20781 raft_consensus.cc:388] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:45:59.678958 20781 raft_consensus.cc:482] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:45:59.679085 20781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:45:59.679946 20781 raft_consensus.cc:504] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b8ebeed3a4f3f8467d506594650d0" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35873 } }
I20260808 04:45:59.680284 20781 leader_election.cc:302] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a91b8ebeed3a4f3f8467d506594650d0; no voters:
I20260808 04:45:59.680595 20781 leader_election.cc:288] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:45:59.680730 20789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:45:59.681015 20789 raft_consensus.cc:686] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [term 1 LEADER]: Becoming Leader. State: Replica: a91b8ebeed3a4f3f8467d506594650d0, State: Running, Role: LEADER
I20260808 04:45:59.681365 20789 consensus_queue.cc:237] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b8ebeed3a4f3f8467d506594650d0" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35873 } }
I20260808 04:45:59.682008 20781 sys_catalog.cc:564] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:45:59.683145 20792 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a91b8ebeed3a4f3f8467d506594650d0. Latest consensus state: current_term: 1 leader_uuid: "a91b8ebeed3a4f3f8467d506594650d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b8ebeed3a4f3f8467d506594650d0" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35873 } } }
I20260808 04:45:59.683192 20790 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a91b8ebeed3a4f3f8467d506594650d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b8ebeed3a4f3f8467d506594650d0" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35873 } } }
I20260808 04:45:59.683314 20792 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:59.683346 20790 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:45:59.683737 20801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:45:59.686909 20801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:45:59.692638 20801 catalog_manager.cc:1347] Generated new cluster ID: 008613e7de5e4f5da246f7899e97fd62
I20260808 04:45:59.692719 20801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:45:59.704768 20801 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:45:59.705096 20801 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:45:59.715073 20801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a91b8ebeed3a4f3f8467d506594650d0: Generated new TSK 0
I20260808 04:45:59.715868 20801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:45:59.723394 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35873
--builtin_ntp_servers=127.4.217.20:37917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:45:59.838714 20823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:45:59.838955 20823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:45:59.838994 20823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:45:59.843055 20823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:45:59.843107 20823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:45:59.843192 20823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:45:59.848445 20823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:45:59.849583 20823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:45:59.851080 20823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:45:59.861129 20829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:59.861166 20823 server_base.cc:1034] running on GCE node
W20260808 04:45:59.861167 20830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:45:59.861361 20832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:45:59.861771 20823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:45:59.862432 20823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:45:59.863623 20823 hybrid_clock.cc:648] HybridClock initialized: now 1786164359863608 us; error 25 us; skew 500 ppm
I20260808 04:45:59.865592 20823 webserver.cc:460] Webserver started at http://127.4.217.1:43905/ using document root <none> and password file <none>
I20260808 04:45:59.866199 20823 fs_manager.cc:359] Metadata directory not provided
I20260808 04:45:59.866297 20823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:45:59.866552 20823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:45:59.868628 20823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "d0122d34372442b18ba0057d33da3695"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "e1d8473ed6754641a6c3f67a370ca7a9"
server_key_iv: "aaca6f093f12980811eb8a2a0749eeb5"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.869297 20823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "d0122d34372442b18ba0057d33da3695"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "e1d8473ed6754641a6c3f67a370ca7a9"
server_key_iv: "aaca6f093f12980811eb8a2a0749eeb5"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.873031 20823 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:45:59.875604 20841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.876928 20823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:45:59.877066 20823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "d0122d34372442b18ba0057d33da3695"
format_stamp: "Formatted at 2026-08-08 04:45:59 on dist-test-slave-2kcd"
server_key: "e1d8473ed6754641a6c3f67a370ca7a9"
server_key_iv: "aaca6f093f12980811eb8a2a0749eeb5"
server_key_version: "encryptionkey@0"
I20260808 04:45:59.877190 20823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:45:59.881852 20823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:45:59.882548 20823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:45:59.882745 20823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:45:59.883323 20823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:45:59.884318 20823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:45:59.884404 20823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.884475 20823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:45:59.884527 20823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:45:59.893513 20823 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:34617
I20260808 04:45:59.893584 21020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:34617 every 8 connection(s)
I20260808 04:45:59.894501 20823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:45:59.900959 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:59.896440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:54639 (local address 127.4.217.62:35873)
0808 04:45:59.896592 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:45:59.896596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:45:59.897685 (+ 1089us) server_negotiation.cc:407] Connection header received
0808 04:45:59.899011 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:45:59.899014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:45:59.899053 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:45:59.899133 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:45:59.900381 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:45:59.900384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:45:59.900386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:45:59.900398 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:45:59.900419 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:45:59.900427 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:45:59.900507 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:45:59.900727 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 04:45:59.900799 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:45:59.900966 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 20823
I20260808 04:45:59.901083 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:45:59.901367 4964 external_mini_cluster.cc:1428] Setting key cbf26d14fc5f6c6b8ce9dc501d268d83
I20260808 04:45:59.901933 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:45:59.896852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35873 (local address 127.4.217.1:54639)
0808 04:45:59.897524 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0808 04:45:59.897575 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:45:59.898786 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:45:59.899340 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:45:59.899347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:45:59.899449 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:45:59.900105 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:45:59.900106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:45:59.900149 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:45:59.900219 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:45:59.900527 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:45:59.900536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:45:59.900547 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 04:45:59.900710 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 04:45:59.900976 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260808 04:45:59.903499 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35873
--builtin_ntp_servers=127.4.217.20:37917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 04:45:59.905292 21021 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35873
I20260808 04:45:59.905565 21021 heartbeater.cc:464] Registering TS with master...
I20260808 04:45:59.906080 21021 heartbeater.cc:511] Master 127.4.217.62:35873 requested a full tablet report, sending...
I20260808 04:45:59.907773 20714 ts_manager.cc:194] Registered new tserver with Master: d0122d34372442b18ba0057d33da3695 (127.4.217.1:34617)
W20260808 04:46:00.017689 21028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:00.017941 21028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:00.017978 21028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:00.021981 21028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:00.022029 21028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:00.022117 21028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:46:00.026837 21028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:00.027869 21028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:00.029368 21028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:00.039117 21035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:00.039201 21028 server_base.cc:1034] running on GCE node
W20260808 04:46:00.039157 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:00.039124 21034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:00.039803 21028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:00.040462 21028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:00.041668 21028 hybrid_clock.cc:648] HybridClock initialized: now 1786164360041655 us; error 25 us; skew 500 ppm
I20260808 04:46:00.043562 21028 webserver.cc:460] Webserver started at http://127.4.217.2:40581/ using document root <none> and password file <none>
I20260808 04:46:00.044153 21028 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:00.044248 21028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:00.044486 21028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:46:00.046588 21028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "4db29abec7b841cf8cb0181ffcc707bf"
format_stamp: "Formatted at 2026-08-08 04:46:00 on dist-test-slave-2kcd"
server_key: "92de828458962aa363693be896e8af30"
server_key_iv: "f2d530615ca87857dfbcd4d3eabe7380"
server_key_version: "encryptionkey@0"
I20260808 04:46:00.047164 21028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "4db29abec7b841cf8cb0181ffcc707bf"
format_stamp: "Formatted at 2026-08-08 04:46:00 on dist-test-slave-2kcd"
server_key: "92de828458962aa363693be896e8af30"
server_key_iv: "f2d530615ca87857dfbcd4d3eabe7380"
server_key_version: "encryptionkey@0"
I20260808 04:46:00.050886 21028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:46:00.053565 21049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:00.054721 21028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:46:00.054859 21028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "4db29abec7b841cf8cb0181ffcc707bf"
format_stamp: "Formatted at 2026-08-08 04:46:00 on dist-test-slave-2kcd"
server_key: "92de828458962aa363693be896e8af30"
server_key_iv: "f2d530615ca87857dfbcd4d3eabe7380"
server_key_version: "encryptionkey@0"
I20260808 04:46:00.054987 21028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:00.059777 21028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:00.060379 21028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:00.060585 21028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:00.061201 21028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:46:00.062101 21028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:46:00.062175 21028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:00.062237 21028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:46:00.062287 21028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:00.071206 21028 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:38973
I20260808 04:46:00.071226 21229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:38973 every 8 connection(s)
I20260808 04:46:00.072091 21028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:46:00.078092 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:00.074063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:54853 (local address 127.4.217.62:35873)
0808 04:46:00.074217 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:00.074220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:00.075255 (+ 1035us) server_negotiation.cc:407] Connection header received
0808 04:46:00.076518 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:00.076521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:00.076562 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:00.076635 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:00.077690 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:00.077693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:00.077695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:46:00.077708 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:00.077728 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:46:00.077736 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:00.077812 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:00.077934 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 04:46:00.077973 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:46:00.078900 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:00.074445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35873 (local address 127.4.217.2:54853)
0808 04:46:00.075114 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:00.075151 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:00.076319 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:00.076843 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:00.076858 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:00.076947 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:46:00.077446 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:46:00.077448 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:46:00.077486 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:46:00.077549 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:46:00.077812 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:46:00.077819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:46:00.077823 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:46:00.077952 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 04:46:00.078155 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260808 04:46:00.080080 21230 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35873
I20260808 04:46:00.080354 21230 heartbeater.cc:464] Registering TS with master...
I20260808 04:46:00.080825 21230 heartbeater.cc:511] Master 127.4.217.62:35873 requested a full tablet report, sending...
I20260808 04:46:00.081061 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 21028
I20260808 04:46:00.081260 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:46:00.081585 4964 external_mini_cluster.cc:1428] Setting key b8f4a8ae72bc0089494311c2bcc2851a
I20260808 04:46:00.082085 20714 ts_manager.cc:194] Registered new tserver with Master: 4db29abec7b841cf8cb0181ffcc707bf (127.4.217.2:38973)
I20260808 04:46:00.083771 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35873
--builtin_ntp_servers=127.4.217.20:37917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:46:00.198436 21235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:00.198693 21235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:00.198760 21235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:00.203028 21235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:00.203114 21235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:00.203220 21235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:46:00.207880 21235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:00.208961 21235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:00.210492 21235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:00.220425 21241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:00.220434 21240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:00.220510 21235 server_base.cc:1034] running on GCE node
W20260808 04:46:00.220427 21243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:00.221112 21235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:00.221715 21235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:00.222921 21235 hybrid_clock.cc:648] HybridClock initialized: now 1786164360222887 us; error 47 us; skew 500 ppm
I20260808 04:46:00.229419 21235 webserver.cc:460] Webserver started at http://127.4.217.3:32895/ using document root <none> and password file <none>
I20260808 04:46:00.230007 21235 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:00.230100 21235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:00.230331 21235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:46:00.232370 21235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "72d96800ef95424384715c27d7fd8e4a"
format_stamp: "Formatted at 2026-08-08 04:46:00 on dist-test-slave-2kcd"
server_key: "3e6288f582a7c681d7d3a70582e38e79"
server_key_iv: "0092d20f4f341827ca3c57d38470a880"
server_key_version: "encryptionkey@0"
I20260808 04:46:00.233068 21235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "72d96800ef95424384715c27d7fd8e4a"
format_stamp: "Formatted at 2026-08-08 04:46:00 on dist-test-slave-2kcd"
server_key: "3e6288f582a7c681d7d3a70582e38e79"
server_key_iv: "0092d20f4f341827ca3c57d38470a880"
server_key_version: "encryptionkey@0"
I20260808 04:46:00.236691 21235 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:46:00.239297 21255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:00.240471 21235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:46:00.240607 21235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "72d96800ef95424384715c27d7fd8e4a"
format_stamp: "Formatted at 2026-08-08 04:46:00 on dist-test-slave-2kcd"
server_key: "3e6288f582a7c681d7d3a70582e38e79"
server_key_iv: "0092d20f4f341827ca3c57d38470a880"
server_key_version: "encryptionkey@0"
I20260808 04:46:00.240718 21235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:00.245322 21235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:00.246026 21235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:00.246213 21235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:00.246750 21235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:46:00.247674 21235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:46:00.247747 21235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:00.247818 21235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:46:00.247869 21235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:00.259382 21235 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:44491
I20260808 04:46:00.259409 21441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:44491 every 8 connection(s)
I20260808 04:46:00.260390 21235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:46:00.266575 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:00.262352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:38931 (local address 127.4.217.62:35873)
0808 04:46:00.262508 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:00.262512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:00.263599 (+ 1087us) server_negotiation.cc:407] Connection header received
0808 04:46:00.264969 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:00.264972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:00.265012 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:00.265095 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:00.266184 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:00.266187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:00.266189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:46:00.266201 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:00.266221 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:46:00.266229 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:00.266308 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:00.266429 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 04:46:00.266468 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:46:00.267483 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:00.262747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35873 (local address 127.4.217.3:38931)
0808 04:46:00.263453 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:00.263493 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:00.264707 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:00.265256 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:00.265262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:00.265347 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:46:00.265894 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:46:00.265896 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:46:00.265934 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:46:00.266000 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:46:00.266307 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:46:00.266314 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:46:00.266319 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:46:00.266449 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 04:46:00.266638 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":117,"threads_started":1}
I20260808 04:46:00.268617 21442 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35873
I20260808 04:46:00.269034 21442 heartbeater.cc:464] Registering TS with master...
I20260808 04:46:00.269434 21442 heartbeater.cc:511] Master 127.4.217.62:35873 requested a full tablet report, sending...
I20260808 04:46:00.270459 20715 ts_manager.cc:194] Registered new tserver with Master: 72d96800ef95424384715c27d7fd8e4a (127.4.217.3:44491)
I20260808 04:46:00.270712 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 21235
I20260808 04:46:00.270854 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:46:00.271188 4964 external_mini_cluster.cc:1428] Setting key 1448a2dfa88decabfdf98d2fa8c9a453
I20260808 04:46:00.273466 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:46:00.276172 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:46:00.275453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.4.217.62:35873)
0808 04:46:00.275571 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:00.275574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:00.275701 (+ 127us) server_negotiation.cc:407] Connection header received
0808 04:46:00.275790 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:00.275792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:00.275822 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:00.275922 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:00.276078 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55224: BlockingRecv error: recv got EOF from 127.0.0.1:55224 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:46:00.276373 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 20823
I20260808 04:46:00.297449 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 21028
I20260808 04:46:00.310596 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 21235
I20260808 04:46:00.328665 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 20666
2026-08-08T04:46:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (864 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:46:00 dist-test-slave-2kcd krb5kdc[21458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:46:00 dist-test-slave-2kcd krb5kdc[21458](info): set up 2 sockets
Aug 08 04:46:00 dist-test-slave-2kcd krb5kdc[21458](info): commencing operation
krb5kdc: starting...
W20260808 04:46:01.606122 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:46:01 dist-test-slave-2kcd krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:46:01.733964 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33307
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:33307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:46:01.846987 21479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:01.847272 21479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:01.847345 21479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:01.851395 21479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:46:01.851478 21479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:01.851513 21479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:01.851554 21479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:46:01.851588 21479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:46:01.856755 21479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:33307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:01.857852 21479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:01.858893 21479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:01.867549 21489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:01.867529 21486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:01.867677 21479 server_base.cc:1034] running on GCE node
W20260808 04:46:01.867533 21487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:01.868436 21479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:01.869568 21479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:01.870735 21479 hybrid_clock.cc:648] HybridClock initialized: now 1786164361870722 us; error 25 us; skew 500 ppm
Aug 08 04:46:01 dist-test-slave-2kcd krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164361, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:46:01.873610 21479 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:46:01.874493 21479 webserver.cc:460] Webserver started at http://127.4.217.62:34115/ using document root <none> and password file <none>
I20260808 04:46:01.875043 21479 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:01.875137 21479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:01.875377 21479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:46:01.877475 21479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "0b92ff774bc04c04b31321816e52608d"
format_stamp: "Formatted at 2026-08-08 04:46:01 on dist-test-slave-2kcd"
server_key: "b6a734c4abafd23845d8b2cfecacf927"
server_key_iv: "a60aae6b12ca4aa29d6c5823f6002333"
server_key_version: "encryptionkey@0"
I20260808 04:46:01.878036 21479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "0b92ff774bc04c04b31321816e52608d"
format_stamp: "Formatted at 2026-08-08 04:46:01 on dist-test-slave-2kcd"
server_key: "b6a734c4abafd23845d8b2cfecacf927"
server_key_iv: "a60aae6b12ca4aa29d6c5823f6002333"
server_key_version: "encryptionkey@0"
I20260808 04:46:01.881626 21479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:46:01.884014 21500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:01.885265 21479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:46:01.885411 21479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "0b92ff774bc04c04b31321816e52608d"
format_stamp: "Formatted at 2026-08-08 04:46:01 on dist-test-slave-2kcd"
server_key: "b6a734c4abafd23845d8b2cfecacf927"
server_key_iv: "a60aae6b12ca4aa29d6c5823f6002333"
server_key_version: "encryptionkey@0"
I20260808 04:46:01.885521 21479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:01.921679 21479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:01.925295 21479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:01.925499 21479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:01.932731 21479 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:33307
I20260808 04:46:01.933034 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:33307 every 8 connection(s)
I20260808 04:46:01.933750 21479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:46:01.936489 21588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:46:01.940430 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 21479
I20260808 04:46:01.940567 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:46:01.940886 4964 external_mini_cluster.cc:1428] Setting key 9c8d1eee8185f8126ff298e5c686d30d
I20260808 04:46:01.942595 21588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d: Bootstrap starting.
I20260808 04:46:01.945123 21588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d: Neither blocks nor log segments found. Creating new log.
I20260808 04:46:01.945861 21588 log.cc:824] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:46:01.948016 21588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d: No bootstrap required, opened a new log
Aug 08 04:46:01 dist-test-slave-2kcd krb5kdc[21458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:46:01.952222 21588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92ff774bc04c04b31321816e52608d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33307 } }
I20260808 04:46:01.952503 21588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:46:01.952577 21588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b92ff774bc04c04b31321816e52608d, State: Initialized, Role: FOLLOWER
I20260808 04:46:01.953104 21588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92ff774bc04c04b31321816e52608d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33307 } }
I20260808 04:46:01.953255 21588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:46:01.953343 21588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:46:01.953451 21588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:46:01.954568 21588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92ff774bc04c04b31321816e52608d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33307 } }
I20260808 04:46:01.954929 21588 leader_election.cc:302] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b92ff774bc04c04b31321816e52608d; no voters:
I20260808 04:46:01.955245 21588 leader_election.cc:288] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:46:01.955358 21595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:46:01.955590 21595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [term 1 LEADER]: Becoming Leader. State: Replica: 0b92ff774bc04c04b31321816e52608d, State: Running, Role: LEADER
I20260808 04:46:01.955910 21595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92ff774bc04c04b31321816e52608d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33307 } }
I20260808 04:46:01.956216 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:01.942275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.4.217.62:33307)
0808 04:46:01.942683 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:01.942722 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:01.942766 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:46:01.943375 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:01.943391 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:01.943711 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:01.943973 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:01.944904 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:01.945964 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:01.947062 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:01.947506 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:01.947635 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:01.950408 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:01.950415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:01.950431 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:01.950461 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:01.953011 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:01.953504 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:01.953510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:01.953517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:01.953663 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:01.953943 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:01.953946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:01.953948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:01.954453 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:01.954663 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:01.954938 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 04:46:01.955242 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":131,"threads_started":1}
I20260808 04:46:01.956497 21588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:46:01.957695 21599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b92ff774bc04c04b31321816e52608d. Latest consensus state: current_term: 1 leader_uuid: "0b92ff774bc04c04b31321816e52608d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92ff774bc04c04b31321816e52608d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33307 } } }
I20260808 04:46:01.957831 21599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [sys.catalog]: This master's current role is: LEADER
I20260808 04:46:01.958153 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b92ff774bc04c04b31321816e52608d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92ff774bc04c04b31321816e52608d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 33307 } } }
I20260808 04:46:01.958266 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d [sys.catalog]: This master's current role is: LEADER
I20260808 04:46:01.958531 21608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:46:01.961640 21608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:46:01.967517 21608 catalog_manager.cc:1347] Generated new cluster ID: 4470d470c8144fd8a56e450c3b815018
I20260808 04:46:01.967592 21608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:46:01.980074 21608 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:46:01.981041 21608 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:46:01.987439 21608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d: Generated new TSK 0
I20260808 04:46:01.988082 21608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:46:02.020090 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33307
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:46:02.132504 21637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:02.132761 21637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:02.132877 21637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:02.137383 21637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:02.137454 21637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:02.137573 21637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:46:02.143015 21637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:02.144086 21637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:02.145227 21637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:02.155681 21646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:02.155686 21648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:02.155758 21650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:02.155970 21637 server_base.cc:1034] running on GCE node
I20260808 04:46:02.156407 21637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:02.157125 21637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:02.158306 21637 hybrid_clock.cc:648] HybridClock initialized: now 1786164362158275 us; error 41 us; skew 500 ppm
Aug 08 04:46:02 dist-test-slave-2kcd krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:46:02.161158 21637 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:46:02.162009 21637 webserver.cc:460] Webserver started at http://127.4.217.1:43575/ using document root <none> and password file <none>
I20260808 04:46:02.162578 21637 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:02.162673 21637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:02.162914 21637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:46:02.164952 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "3e9b287d975f4f7b8f2bde580c05a430"
format_stamp: "Formatted at 2026-08-08 04:46:02 on dist-test-slave-2kcd"
server_key: "c24f1f2add835423574a0e50ebad24d4"
server_key_iv: "3a407e200813347d17cb3b649c8c9623"
server_key_version: "encryptionkey@0"
I20260808 04:46:02.165524 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "3e9b287d975f4f7b8f2bde580c05a430"
format_stamp: "Formatted at 2026-08-08 04:46:02 on dist-test-slave-2kcd"
server_key: "c24f1f2add835423574a0e50ebad24d4"
server_key_iv: "3a407e200813347d17cb3b649c8c9623"
server_key_version: "encryptionkey@0"
I20260808 04:46:02.169570 21637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260808 04:46:02.171867 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:02.173133 21637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:46:02.173270 21637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "3e9b287d975f4f7b8f2bde580c05a430"
format_stamp: "Formatted at 2026-08-08 04:46:02 on dist-test-slave-2kcd"
server_key: "c24f1f2add835423574a0e50ebad24d4"
server_key_iv: "3a407e200813347d17cb3b649c8c9623"
server_key_version: "encryptionkey@0"
I20260808 04:46:02.173388 21637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:02.188496 21637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:02.191318 21637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:02.191526 21637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:02.192080 21637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:46:02.192932 21637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:46:02.193001 21637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:02.193069 21637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:46:02.193116 21637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:02.202015 21637 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:33861
I20260808 04:46:02.202064 21829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:33861 every 8 connection(s)
I20260808 04:46:02.202862 21637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:46:02.206684 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 21637
I20260808 04:46:02.206810 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:46:02.207084 4964 external_mini_cluster.cc:1428] Setting key e8653500f7a97e097d60247ac1870efe
Aug 08 04:46:02 dist-test-slave-2kcd krb5kdc[21458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:46:02.217413 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.204637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:41541 (local address 127.4.217.62:33307)
0808 04:46:02.204769 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:02.204772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:02.205812 (+ 1040us) server_negotiation.cc:407] Connection header received
0808 04:46:02.206705 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:02.206709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:02.206766 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:02.206852 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:02.208214 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.209022 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.210140 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.210547 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.210690 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.213491 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:02.213494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:02.213496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:02.213518 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:02.215260 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:02.215815 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:02.215819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:02.215823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:02.215916 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:02.216268 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:02.216273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:02.216275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:02.216458 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:02.216569 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:02.217109 (+ 540us) server_negotiation.cc:299] Negotiation successful
0808 04:46:02.217234 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
I20260808 04:46:02.218166 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.205032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33307 (local address 127.4.217.1:41541)
0808 04:46:02.205663 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:02.205702 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:02.206504 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:02.207017 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:02.207023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:02.207448 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:46:02.208064 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.208087 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.209164 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.209166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.210011 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.210021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.210763 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.210768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.210998 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.211590 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:46:02.211614 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:46:02.213310 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:46:02.215411 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:02.215417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:02.215431 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:02.215695 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:02.216017 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:02.216021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:02.216023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:02.216166 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:02.216569 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:46:02.216577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:46:02.216868 (+ 291us) client_negotiation.cc:769] Sending connection context
0808 04:46:02.217063 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 04:46:02.217384 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":102,"threads_started":1}
I20260808 04:46:02.219400 21832 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33307
I20260808 04:46:02.219651 21832 heartbeater.cc:464] Registering TS with master...
I20260808 04:46:02.220134 21832 heartbeater.cc:511] Master 127.4.217.62:33307 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:46:02.221978 21529 ts_manager.cc:194] Registered new tserver with Master: 3e9b287d975f4f7b8f2bde580c05a430 (127.4.217.1:33861)
I20260808 04:46:02.223151 21529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:41541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:46:02.235605 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33307
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:46:02.350908 21838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:02.351171 21838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:02.351260 21838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:02.355161 21838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:02.355244 21838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:02.355329 21838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:46:02.360729 21838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:02.361726 21838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:02.362722 21838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:02.372197 21853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:02.372236 21846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:02.372262 21849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:02.372689 21838 server_base.cc:1034] running on GCE node
I20260808 04:46:02.373126 21838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:02.373755 21838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:02.374919 21838 hybrid_clock.cc:648] HybridClock initialized: now 1786164362374903 us; error 26 us; skew 500 ppm
Aug 08 04:46:02 dist-test-slave-2kcd krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:46:02.377800 21838 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:46:02.378582 21838 webserver.cc:460] Webserver started at http://127.4.217.2:42769/ using document root <none> and password file <none>
I20260808 04:46:02.379088 21838 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:02.379145 21838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:02.379311 21838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:46:02.381315 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "1f5f9f658fa848f9a30dc856fdbac9dc"
format_stamp: "Formatted at 2026-08-08 04:46:02 on dist-test-slave-2kcd"
server_key: "7c043fb632e885c06bf6992b0567d09a"
server_key_iv: "89d1c325f65828ff146d1f76662b9d34"
server_key_version: "encryptionkey@0"
I20260808 04:46:02.381822 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "1f5f9f658fa848f9a30dc856fdbac9dc"
format_stamp: "Formatted at 2026-08-08 04:46:02 on dist-test-slave-2kcd"
server_key: "7c043fb632e885c06bf6992b0567d09a"
server_key_iv: "89d1c325f65828ff146d1f76662b9d34"
server_key_version: "encryptionkey@0"
I20260808 04:46:02.385458 21838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:46:02.387732 21863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:02.389053 21838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:46:02.389214 21838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "1f5f9f658fa848f9a30dc856fdbac9dc"
format_stamp: "Formatted at 2026-08-08 04:46:02 on dist-test-slave-2kcd"
server_key: "7c043fb632e885c06bf6992b0567d09a"
server_key_iv: "89d1c325f65828ff146d1f76662b9d34"
server_key_version: "encryptionkey@0"
I20260808 04:46:02.389305 21838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:02.412108 21838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:02.415189 21838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:02.415335 21838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:02.417984 21838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:46:02.418797 21838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:46:02.418844 21838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:02.418881 21838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:46:02.418896 21838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:02.428112 21838 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:45365
I20260808 04:46:02.428171 22041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:45365 every 8 connection(s)
I20260808 04:46:02.429203 21838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:46:02.432317 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 21838
I20260808 04:46:02.432451 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:46:02.432802 4964 external_mini_cluster.cc:1428] Setting key 562e159c18c2afea41dcb3012f4dfab0
Aug 08 04:46:02 dist-test-slave-2kcd krb5kdc[21458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:46:02.445331 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.431282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:33861 (local address 127.4.217.62:33307)
0808 04:46:02.431447 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:02.431452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:02.432550 (+ 1098us) server_negotiation.cc:407] Connection header received
0808 04:46:02.434248 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:02.434251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:02.434307 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:02.434398 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:02.435970 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.436514 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.437924 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.438165 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.438280 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.441237 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:02.441240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:02.441242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:02.441267 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:02.442733 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:02.443391 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:02.443394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:02.443395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:02.443509 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:02.443946 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:02.443950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:02.443951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:02.444181 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:02.444267 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:02.445009 (+ 742us) server_negotiation.cc:299] Negotiation successful
0808 04:46:02.445151 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:46:02.446249 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.431686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33307 (local address 127.4.217.2:33861)
0808 04:46:02.432379 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:02.432449 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:02.434001 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:02.434599 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:02.434608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:02.435118 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:46:02.435817 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.435840 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.436649 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.436653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.437790 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.437801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.438323 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.438328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.438512 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.439149 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:46:02.439166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:46:02.441057 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:46:02.442873 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:02.442881 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:02.442898 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:02.443284 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:02.443643 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:02.443650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:02.443652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:02.443835 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:02.444336 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:46:02.444342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:46:02.444657 (+ 315us) client_negotiation.cc:769] Sending connection context
0808 04:46:02.444875 (+ 218us) client_negotiation.cc:240] Negotiation successful
0808 04:46:02.445231 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
I20260808 04:46:02.447578 22042 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33307
I20260808 04:46:02.447819 22042 heartbeater.cc:464] Registering TS with master...
I20260808 04:46:02.448354 22042 heartbeater.cc:511] Master 127.4.217.62:33307 requested a full tablet report, sending...
I20260808 04:46:02.449640 21529 ts_manager.cc:194] Registered new tserver with Master: 1f5f9f658fa848f9a30dc856fdbac9dc (127.4.217.2:45365)
I20260808 04:46:02.450238 21529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:33861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:46:02.464023 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33307
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:46:02.575563 22047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:02.575784 22047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:02.575824 22047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:02.579787 22047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:02.579836 22047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:02.579895 22047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:46:02.584807 22047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:33307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:02.585680 22047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:02.586577 22047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:02.596381 22059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:02.596436 22056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:02.596521 22047 server_base.cc:1034] running on GCE node
W20260808 04:46:02.596423 22057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:02.597146 22047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:02.597741 22047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:02.598906 22047 hybrid_clock.cc:648] HybridClock initialized: now 1786164362598887 us; error 30 us; skew 500 ppm
Aug 08 04:46:02 dist-test-slave-2kcd krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:46:02.601552 22047 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:46:02.602413 22047 webserver.cc:460] Webserver started at http://127.4.217.3:33765/ using document root <none> and password file <none>
I20260808 04:46:02.602981 22047 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:02.603045 22047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:02.603276 22047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:46:02.605352 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "2d51417c54f143868119045edb3528e1"
format_stamp: "Formatted at 2026-08-08 04:46:02 on dist-test-slave-2kcd"
server_key: "def858c5de5d94bfa8ee82760ba036bb"
server_key_iv: "9f69ef2ce492b017744f8d056b9c7381"
server_key_version: "encryptionkey@0"
I20260808 04:46:02.606006 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "2d51417c54f143868119045edb3528e1"
format_stamp: "Formatted at 2026-08-08 04:46:02 on dist-test-slave-2kcd"
server_key: "def858c5de5d94bfa8ee82760ba036bb"
server_key_iv: "9f69ef2ce492b017744f8d056b9c7381"
server_key_version: "encryptionkey@0"
I20260808 04:46:02.609769 22047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:46:02.612128 22070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:02.613363 22047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:46:02.613514 22047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "2d51417c54f143868119045edb3528e1"
format_stamp: "Formatted at 2026-08-08 04:46:02 on dist-test-slave-2kcd"
server_key: "def858c5de5d94bfa8ee82760ba036bb"
server_key_iv: "9f69ef2ce492b017744f8d056b9c7381"
server_key_version: "encryptionkey@0"
I20260808 04:46:02.613631 22047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:02.640599 22047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:02.643778 22047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:02.644003 22047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:02.644659 22047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:46:02.645566 22047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:46:02.645643 22047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:02.645717 22047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:46:02.645762 22047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:02.655350 22047 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:38177
I20260808 04:46:02.655376 22261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:38177 every 8 connection(s)
I20260808 04:46:02.656239 22047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:46:02.660714 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 22047
I20260808 04:46:02.660876 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:46:02.661203 4964 external_mini_cluster.cc:1428] Setting key f4d272eff477be9582c4a85c218a1c91
Aug 08 04:46:02 dist-test-slave-2kcd krb5kdc[21458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:46:02.670037 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.658261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:45649 (local address 127.4.217.62:33307)
0808 04:46:02.658433 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:02.658438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:02.659375 (+ 937us) server_negotiation.cc:407] Connection header received
0808 04:46:02.660841 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:02.660846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:02.660934 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:02.661031 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:02.662453 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.662981 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.664102 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.664324 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.664433 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.666794 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:02.666798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:02.666800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:02.666824 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:02.668025 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:02.668525 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:02.668533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:02.668534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:02.668621 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:02.669083 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:02.669087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:02.669088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:02.669293 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:02.669379 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:02.669816 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 04:46:02.669918 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:46:02.670779 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.658729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:33307 (local address 127.4.217.3:45649)
0808 04:46:02.659218 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:02.659262 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:02.660544 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:02.661228 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:02.661235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:02.661629 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:46:02.662305 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.662321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.663110 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.663113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.663974 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.663984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.664474 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.664479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.664647 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.665225 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:46:02.665241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:46:02.666619 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:46:02.668147 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:02.668153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:02.668167 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:02.668409 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:02.668716 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:02.668720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:02.668722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:02.668941 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:02.669393 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:46:02.669397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:46:02.669614 (+ 217us) client_negotiation.cc:769] Sending connection context
0808 04:46:02.669763 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:46:02.669997 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260808 04:46:02.671851 22262 heartbeater.cc:347] Connected to a master server at 127.4.217.62:33307
I20260808 04:46:02.672066 22262 heartbeater.cc:464] Registering TS with master...
I20260808 04:46:02.672480 22262 heartbeater.cc:511] Master 127.4.217.62:33307 requested a full tablet report, sending...
I20260808 04:46:02.673509 21529 ts_manager.cc:194] Registered new tserver with Master: 2d51417c54f143868119045edb3528e1 (127.4.217.3:38177)
I20260808 04:46:02.674198 21529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:45649
I20260808 04:46:02.675123 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:46:02.684460 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.677106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.4.217.62:33307)
0808 04:46:02.677218 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:02.677222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:02.677457 (+ 235us) server_negotiation.cc:407] Connection header received
0808 04:46:02.677526 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:02.677528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:02.677563 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:02.677624 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:02.678253 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.678743 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.679604 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.679832 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.679942 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.681353 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:02.681356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:02.681359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:02.681383 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:02.682770 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:02.683211 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:02.683214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:02.683216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:02.683293 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:02.683573 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:02.683576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:02.683578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:02.683778 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:02.683935 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:02.684194 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 04:46:02.684305 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:46:02.687261 21529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:46:02.690407 21529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:46:02.704720 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.698794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:38177 (local address 127.0.0.1:37718)
0808 04:46:02.699159 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:02.699211 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:02.699336 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:02.700112 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:02.700118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:02.700146 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:02.700318 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.700326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.701707 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.701711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.703025 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.703037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.704148 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.704152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.704269 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.704316 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 04:46:02.704435 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 04:46:02.704565 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"spinlock_wait_cycles":6400,"thread_start_us":92,"threads_started":1}
I20260808 04:46:02.704727 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.698911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.4.217.3:38177)
0808 04:46:02.699306 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:02.699318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:02.699344 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:46:02.699541 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:02.699548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:02.699751 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:02.699910 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:02.700458 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.701564 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.703178 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.704030 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.704132 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.704275 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:02.704484 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 04:46:02.704580 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":128,"threads_started":1}
I20260808 04:46:02.705268 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.699516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:45365 (local address 127.0.0.1:54022)
0808 04:46:02.700085 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:02.700099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:02.700207 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:02.701131 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:02.701134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:02.701154 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:02.701246 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.701252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.702621 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.702624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.703595 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.703604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.704846 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.704852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.704951 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.704976 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:46:02.705065 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 04:46:02.705134 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":32,"spinlock_wait_cycles":1536,"thread_start_us":143,"threads_started":1}
I20260808 04:46:02.705739 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.699688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.4.217.2:45365)
0808 04:46:02.700124 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:02.700135 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:02.700160 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:46:02.700758 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:02.700764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:02.700972 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:02.701136 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:02.701343 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.702455 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.703718 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.704530 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.704629 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.704758 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:02.705496 (+ 738us) server_negotiation.cc:299] Negotiation successful
0808 04:46:02.705589 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":124,"threads_started":1}
I20260808 04:46:02.706180 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.700386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.4.217.1:33861)
0808 04:46:02.700688 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:02.700705 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:02.700725 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:46:02.700818 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:02.700822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:02.700972 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:02.701131 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:02.701547 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.702508 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.704010 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.705046 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.705183 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.705317 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:02.705935 (+ 618us) server_negotiation.cc:299] Negotiation successful
0808 04:46:02.706035 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":134,"threads_started":1}
I20260808 04:46:02.706497 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:02.699900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33861 (local address 127.0.0.1:59810)
0808 04:46:02.700435 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:02.700448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:02.700532 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:02.701296 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:02.701298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:02.701321 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:02.701410 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.701416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.702659 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.702662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.703855 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:02.703867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:02.705181 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:02.705185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:02.705613 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:02.705636 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:46:02.705720 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:46:02.706360 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":24,"thread_start_us":113,"threads_started":1}
I20260808 04:46:02.708163 22157 tablet_service.cc:1467] Processing CreateTablet for tablet f6c273592a4a4f8a9ba8e42a2e45a817 (DEFAULT_TABLE table=test-table [id=3715dc42aa634eb09315bc397fcf5f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:46:02.708498 21933 tablet_service.cc:1467] Processing CreateTablet for tablet f6c273592a4a4f8a9ba8e42a2e45a817 (DEFAULT_TABLE table=test-table [id=3715dc42aa634eb09315bc397fcf5f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:46:02.709278 22157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c273592a4a4f8a9ba8e42a2e45a817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:46:02.709594 21933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c273592a4a4f8a9ba8e42a2e45a817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:46:02.709210 21736 tablet_service.cc:1467] Processing CreateTablet for tablet f6c273592a4a4f8a9ba8e42a2e45a817 (DEFAULT_TABLE table=test-table [id=3715dc42aa634eb09315bc397fcf5f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:46:02.710165 21736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c273592a4a4f8a9ba8e42a2e45a817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:46:02.715633 22290 tablet_bootstrap.cc:492] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: Bootstrap starting.
I20260808 04:46:02.717126 22291 tablet_bootstrap.cc:492] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: Bootstrap starting.
I20260808 04:46:02.718211 22290 tablet_bootstrap.cc:654] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: Neither blocks nor log segments found. Creating new log.
I20260808 04:46:02.719668 22290 log.cc:824] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:46:02.719843 22291 tablet_bootstrap.cc:654] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: Neither blocks nor log segments found. Creating new log.
I20260808 04:46:02.720382 22293 tablet_bootstrap.cc:492] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430: Bootstrap starting.
I20260808 04:46:02.720727 22291 log.cc:824] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:46:02.721988 22290 tablet_bootstrap.cc:492] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: No bootstrap required, opened a new log
I20260808 04:46:02.722193 22290 ts_tablet_manager.cc:1397] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:46:02.722476 22291 tablet_bootstrap.cc:492] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: No bootstrap required, opened a new log
I20260808 04:46:02.722677 22291 ts_tablet_manager.cc:1397] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:46:02.722677 22293 tablet_bootstrap.cc:654] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430: Neither blocks nor log segments found. Creating new log.
I20260808 04:46:02.723496 22293 log.cc:824] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430: Log is configured to *not* fsync() on all Append() calls
I20260808 04:46:02.725430 22293 tablet_bootstrap.cc:492] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430: No bootstrap required, opened a new log
I20260808 04:46:02.725612 22293 ts_tablet_manager.cc:1397] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:46:02.725818 22291 raft_consensus.cc:348] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } }
I20260808 04:46:02.726051 22291 raft_consensus.cc:374] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:46:02.726114 22291 raft_consensus.cc:729] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5f9f658fa848f9a30dc856fdbac9dc, State: Initialized, Role: FOLLOWER
I20260808 04:46:02.726109 22290 raft_consensus.cc:348] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } }
I20260808 04:46:02.726356 22290 raft_consensus.cc:374] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:46:02.726433 22290 raft_consensus.cc:729] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d51417c54f143868119045edb3528e1, State: Initialized, Role: FOLLOWER
I20260808 04:46:02.726668 22291 consensus_queue.cc:260] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } }
I20260808 04:46:02.726945 22290 consensus_queue.cc:260] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } }
I20260808 04:46:02.727648 22291 ts_tablet_manager.cc:1428] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260808 04:46:02.727917 22290 ts_tablet_manager.cc:1428] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:46:02.728025 22042 heartbeater.cc:503] Master 127.4.217.62:33307 was elected leader, sending a full tablet report...
I20260808 04:46:02.728236 22262 heartbeater.cc:503] Master 127.4.217.62:33307 was elected leader, sending a full tablet report...
I20260808 04:46:02.729766 22293 raft_consensus.cc:348] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } }
I20260808 04:46:02.730014 22293 raft_consensus.cc:374] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:46:02.730073 22293 raft_consensus.cc:729] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9b287d975f4f7b8f2bde580c05a430, State: Initialized, Role: FOLLOWER
I20260808 04:46:02.730474 22293 consensus_queue.cc:260] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } }
I20260808 04:46:02.731068 21832 heartbeater.cc:503] Master 127.4.217.62:33307 was elected leader, sending a full tablet report...
I20260808 04:46:02.731400 22293 ts_tablet_manager.cc:1428] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260808 04:46:02.907363 22264 tablet.cc:2369] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:46:02.931404 22043 tablet.cc:2369] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:46:02.954170 21834 tablet.cc:2369] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:46:03.072479 22300 raft_consensus.cc:482] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:46:03.072731 22300 raft_consensus.cc:504] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } }
I20260808 04:46:03.074064 22300 leader_election.cc:288] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d51417c54f143868119045edb3528e1 (127.4.217.3:38177), 1f5f9f658fa848f9a30dc856fdbac9dc (127.4.217.2:45365)
I20260808 04:46:03.078660 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:03.074292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:38177 (local address 127.4.217.1:40561)
0808 04:46:03.074576 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:03.074592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:03.074712 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:03.075048 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:03.075051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:03.075074 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:03.075206 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:03.075211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.076256 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.076260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:03.077415 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:03.077425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.078181 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.078187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:03.078315 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:03.078339 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:46:03.078420 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 04:46:03.078486 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
I20260808 04:46:03.078679 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:03.074434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:40561 (local address 127.4.217.3:38177)
0808 04:46:03.074576 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:03.074581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:03.074687 (+ 106us) server_negotiation.cc:407] Connection header received
0808 04:46:03.074836 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:03.074839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:03.074895 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:03.074999 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:03.075421 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.076154 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.077567 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.078033 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.078115 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:03.078169 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:03.078450 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 04:46:03.078540 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:46:03.078755 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:03.074893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:33159 (local address 127.4.217.2:45365)
0808 04:46:03.075067 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:03.075072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:03.075349 (+ 277us) server_negotiation.cc:407] Connection header received
0808 04:46:03.075420 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:03.075423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:03.075466 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:03.075525 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:03.075833 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.076500 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.077719 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.078131 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.078258 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:03.078350 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:03.078564 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 04:46:03.078631 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:46:03.079202 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:03.074713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:45365 (local address 127.4.217.1:33159)
0808 04:46:03.075187 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:03.075199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:03.075320 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:03.075559 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:03.075561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:03.075580 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:03.075690 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:03.075696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.076609 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.076611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:03.077602 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:03.077613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.078309 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.078312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:03.078423 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:03.078448 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:46:03.078527 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 04:46:03.078589 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":110,"threads_started":1}
I20260808 04:46:03.079425 22187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c273592a4a4f8a9ba8e42a2e45a817" candidate_uuid: "3e9b287d975f4f7b8f2bde580c05a430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d51417c54f143868119045edb3528e1" is_pre_election: true
I20260808 04:46:03.079785 22187 raft_consensus.cc:2454] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e9b287d975f4f7b8f2bde580c05a430 in term 0.
I20260808 04:46:03.080014 21971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c273592a4a4f8a9ba8e42a2e45a817" candidate_uuid: "3e9b287d975f4f7b8f2bde580c05a430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" is_pre_election: true
I20260808 04:46:03.080305 21971 raft_consensus.cc:2454] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e9b287d975f4f7b8f2bde580c05a430 in term 0.
I20260808 04:46:03.080289 21660 leader_election.cc:302] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d51417c54f143868119045edb3528e1, 3e9b287d975f4f7b8f2bde580c05a430; no voters:
I20260808 04:46:03.080587 22300 raft_consensus.cc:2781] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:46:03.080682 22300 raft_consensus.cc:482] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:46:03.080737 22300 raft_consensus.cc:3037] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:46:03.081889 22300 raft_consensus.cc:504] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } }
I20260808 04:46:03.082226 22300 leader_election.cc:288] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [CANDIDATE]: Term 1 election: Requested vote from peers 2d51417c54f143868119045edb3528e1 (127.4.217.3:38177), 1f5f9f658fa848f9a30dc856fdbac9dc (127.4.217.2:45365)
I20260808 04:46:03.082588 22187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c273592a4a4f8a9ba8e42a2e45a817" candidate_uuid: "3e9b287d975f4f7b8f2bde580c05a430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d51417c54f143868119045edb3528e1"
I20260808 04:46:03.082687 21971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c273592a4a4f8a9ba8e42a2e45a817" candidate_uuid: "3e9b287d975f4f7b8f2bde580c05a430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc"
I20260808 04:46:03.082691 22187 raft_consensus.cc:3037] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:46:03.082821 21971 raft_consensus.cc:3037] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:46:03.083808 22187 raft_consensus.cc:2454] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e9b287d975f4f7b8f2bde580c05a430 in term 1.
I20260808 04:46:03.083936 21971 raft_consensus.cc:2454] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e9b287d975f4f7b8f2bde580c05a430 in term 1.
I20260808 04:46:03.084095 21660 leader_election.cc:302] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d51417c54f143868119045edb3528e1, 3e9b287d975f4f7b8f2bde580c05a430; no voters:
I20260808 04:46:03.084287 22300 raft_consensus.cc:2781] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:46:03.084518 22300 raft_consensus.cc:686] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [term 1 LEADER]: Becoming Leader. State: Replica: 3e9b287d975f4f7b8f2bde580c05a430, State: Running, Role: LEADER
I20260808 04:46:03.084901 22300 consensus_queue.cc:237] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } }
I20260808 04:46:03.088277 21527 catalog_manager.cc:5495] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e9b287d975f4f7b8f2bde580c05a430 (127.4.217.1). New cstate: current_term: 1 leader_uuid: "3e9b287d975f4f7b8f2bde580c05a430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e9b287d975f4f7b8f2bde580c05a430" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33861 } health_report { overall_health: HEALTHY } } }
I20260808 04:46:03.155691 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:03.152096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.4.217.1:33861)
0808 04:46:03.152274 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:03.152281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:03.152304 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:46:03.152384 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:03.152386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:03.152447 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:03.152564 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:46:03.152933 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.153465 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.154365 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:03.154578 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:03.154660 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:03.154958 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:46:03.155379 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:46:03.155471 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:03.155529 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 04:46:03.155580 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:46:03.161902 22187 raft_consensus.cc:1261] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [term 1 FOLLOWER]: Refusing update from remote peer 3e9b287d975f4f7b8f2bde580c05a430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:46:03.161902 21971 raft_consensus.cc:1261] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [term 1 FOLLOWER]: Refusing update from remote peer 3e9b287d975f4f7b8f2bde580c05a430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:46:03.162621 22300 consensus_queue.cc:1035] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5f9f658fa848f9a30dc856fdbac9dc" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 45365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:46:03.162802 22305 consensus_queue.cc:1035] T f6c273592a4a4f8a9ba8e42a2e45a817 P 3e9b287d975f4f7b8f2bde580c05a430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d51417c54f143868119045edb3528e1" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 38177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:46:03.170260 22312 mvcc.cc:205] Tried to move back new op lower bound from 7316129231503589376 to 7316129231198498816. Current Snapshot: MvccSnapshot[applied={T|T < 7316129231503589376}]
I20260808 04:46:03.178181 21528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:46:03.178325 21528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:46:03.180745 21528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b92ff774bc04c04b31321816e52608d: Sending DeleteTablet for 3 replicas of tablet f6c273592a4a4f8a9ba8e42a2e45a817
I20260808 04:46:03.181643 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c273592a4a4f8a9ba8e42a2e45a817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:46:03 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:59810
I20260808 04:46:03.181658 22157 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c273592a4a4f8a9ba8e42a2e45a817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:46:03 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:37718
I20260808 04:46:03.181691 21933 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c273592a4a4f8a9ba8e42a2e45a817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:46:03 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:54022
I20260808 04:46:03.182132 22325 tablet_replica.cc:330] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: stopping tablet replica
I20260808 04:46:03.182488 22325 raft_consensus.cc:2229] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:46:03.182663 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 21637
I20260808 04:46:03.193804 22325 raft_consensus.cc:2258] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:46:03.194370 22323 tablet_replica.cc:330] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: stopping tablet replica
I20260808 04:46:03.195627 22323 raft_consensus.cc:2229] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:46:03.195971 22323 raft_consensus.cc:2258] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:46:03.196484 22325 ts_tablet_manager.cc:1905] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:46:03.197486 22323 ts_tablet_manager.cc:1905] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:46:03.198770 22325 ts_tablet_manager.cc:1918] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:46:03.198890 22325 log.cc:1196] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/f6c273592a4a4f8a9ba8e42a2e45a817
I20260808 04:46:03.199167 22325 ts_tablet_manager.cc:1939] T f6c273592a4a4f8a9ba8e42a2e45a817 P 2d51417c54f143868119045edb3528e1: Deleting consensus metadata
I20260808 04:46:03.200270 21503 catalog_manager.cc:4841] TS 2d51417c54f143868119045edb3528e1 (127.4.217.3:38177): tablet f6c273592a4a4f8a9ba8e42a2e45a817 (table test-table [id=3715dc42aa634eb09315bc397fcf5f11]) successfully deleted
I20260808 04:46:03.200929 22323 ts_tablet_manager.cc:1918] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:46:03.201036 22323 log.cc:1196] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/f6c273592a4a4f8a9ba8e42a2e45a817
I20260808 04:46:03.201365 22323 ts_tablet_manager.cc:1939] T f6c273592a4a4f8a9ba8e42a2e45a817 P 1f5f9f658fa848f9a30dc856fdbac9dc: Deleting consensus metadata
I20260808 04:46:03.202627 21503 catalog_manager.cc:4841] TS 1f5f9f658fa848f9a30dc856fdbac9dc (127.4.217.2:45365): tablet f6c273592a4a4f8a9ba8e42a2e45a817 (table test-table [id=3715dc42aa634eb09315bc397fcf5f11]) successfully deleted
W20260808 04:46:03.203394 21503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:33861 (error 108)
I20260808 04:46:03.203610 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 21838
I20260808 04:46:03.205796 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:03.205524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33861 (local address 127.0.0.1:59830)
0808 04:46:03.205659 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:03.205724 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:46:03.206049 21503 catalog_manager.cc:4584] TS 3e9b287d975f4f7b8f2bde580c05a430 (127.4.217.1:33861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6c273592a4a4f8a9ba8e42a2e45a817: Network error: Client connection negotiation failed: client connection to 127.4.217.1:33861: connect: Connection refused (error 111)
I20260808 04:46:03.214090 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 22047
I20260808 04:46:03.221925 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 21479
2026-08-08T04:46:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2893 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:46:03 dist-test-slave-2kcd krb5kdc[22335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:46:03 dist-test-slave-2kcd krb5kdc[22335](info): set up 2 sockets
Aug 08 04:46:03 dist-test-slave-2kcd krb5kdc[22335](info): commencing operation
krb5kdc: starting...
W20260808 04:46:04.501300 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:46:04 dist-test-slave-2kcd krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:46:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/rangeradmin_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/rangeradmin_127.4.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/rangerlookup_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/rangerlookup_127.4.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/HTTP_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/HTTP_127.4.217.20@KRBTEST.COM.keytab.
I20260808 04:46:04.676941 4964 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskywCe24/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/postgres -l logfile start
2026-08-08 04:46:07.128 UTC [22382] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 04:46:07.128 UTC [22382] LOG: listening on IPv4 address "127.4.217.20", port 34337
2026-08-08 04:46:07.132 UTC [22382] LOG: listening on Unix socket "/tmp/.s.PGSQL.34337"
2026-08-08 04:46:07.170 UTC [22387] LOG: database system was shut down at 2026-08-08 04:46:05 UTC
2026-08-08 04:46:07.174 UTC [22382] LOG: database system is ready to accept connections
I20260808 04:46:08.325367 4964 mini_postgres.cc:96] Postgres bound to 34337
2026-08-08 04:46:08.334 UTC [22401] FATAL: database "slave" does not exist
127.4.217.20:34337 - accepting connections
I20260808 04:46:08.334947 4964 mini_ranger.cc:161] Starting Ranger...
I20260808 04:46:08.351210 4964 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 04:46:08.797773 4964 mini_ranger.cc:87] Created ranger Postgres database
I20260808 04:46:08.797889 4964 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-admin
2026-08-08 04:46:09,148 [I] DB FLAVOR :POSTGRES
2026-08-08 04:46:09,148 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:46:09,148 [I] Checking connection..
2026-08-08 04:46:09,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 04:46:09,415 [I] Checking connection passed.
2026-08-08 04:46:09,415 [I] --------- Verifying version history table ---------
2026-08-08 04:46:09,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:46:09,677 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:46:09,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:46:09,944 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:46:09,944 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 04:46:09,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 04:46:10.192 UTC [22507] WARNING: there is no transaction in progress
2026-08-08 04:46:10,200 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 04:46:10,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:46:10,455 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 04:46:10,456 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:46:10,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 04:46:10,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 04:46:10,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2kcd.c.gcp-upstream.internal','N') ;"
2026-08-08 04:46:11,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:46:11,492 [I] Table x_portal_user does not exist in database ranger
2026-08-08 04:46:11,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:46:11,760 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 04:46:11,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:46:12,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:46:12,284 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 04:46:12,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 04:46:13.231 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.247 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.262 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.276 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.292 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.309 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.363 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.370 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.383 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.393 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.404 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.418 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.428 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.435 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.442 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:13.448 UTC [22681] WARNING: there is no transaction in progress
2026-08-08 04:46:14,075 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 04:46:14,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 04:46:14,324 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 04:46:14,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:46:14,593 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 04:46:14,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:46:14,856 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 04:46:14,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:46:15,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:46:15,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2kcd.c.gcp-upstream.internal';"
2026-08-08 04:46:15,605 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 04:46:15,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 04:46:15,847 [I] DB_PATCHES have already been applied
I20260808 04:46:15.853199 4964 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-admin:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskywCe24/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 04:46:15.853260 4964 mini_ranger.cc:193] Using host: 127.4.217.20
I20260808 04:46:15.857599 4964 mini_ranger.cc:227] Ranger admin URL: http://127.4.217.20:38953
Aug 08, 2026 4:46:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:46:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:46:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 4:46:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 4:46:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 4:46:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 4:46:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 04:46:17.242446 4964 mini_ranger.cc:160] Time spent starting Ranger: real 8.908s user 0.006s sys 0.006s
Aug 08, 2026 4:46:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 4:46:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 08, 2026 4:46:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 4:46:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 4:46:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 04:46:38.340076 4964 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/rangerkms_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/rangerkms_127.4.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.217.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.217.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.217.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/HTTP_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.217.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/HTTP_127.4.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 04:46:38 dist-test-slave-2kcd krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164398, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 04:46:38.432758 4964 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 04:46:38.444222 4964 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 04:46:39.251793 4964 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 04:46:39.251909 4964 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-kms
I20260808 04:46:39.252022 4964 mini_ranger_kms.cc:227] Using postgres at 127.4.217.20:34337
2026-08-08 04:46:39,572 [I] DB FLAVOR :POSTGRES
2026-08-08 04:46:39,572 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:46:39,573 [I] Checking connection
2026-08-08 04:46:39,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 04:46:39,848 [I] connection success
2026-08-08 04:46:39,848 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 04:46:39,848 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 04:46:39,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 04:46:40,080 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 04:46:40,081 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:46:40,081 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 04:46:40,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:34337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 04:46:40,374 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 04:46:40.843770 4964 mini_ranger_kms.cc:310] Created kms service
I20260808 04:46:41.055213 4964 mini_ranger_kms.cc:326] Created kudu user
I20260808 04:46:41.101871 4964 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 04:46:41 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164398, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:46:41.556489 4964 mini_ranger_kms.cc:384] Added ranger policy
I20260808 04:46:41.556876 4964 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-kms:/tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 04:46:41.556924 4964 mini_ranger_kms.cc:240] Using host: 127.4.217.20
I20260808 04:46:41.562170 4964 mini_ranger_kms.cc:276] Ranger KMS PID: 22991
I20260808 04:46:41.562317 4964 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.217.20:45599
Aug 08, 2026 4:46:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:46:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:46:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 4:46:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 4:46:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 4:46:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 04:46:44 dist-test-slave-2kcd krb5kdc[22335](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164404, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.217.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:46:44.426678 4964 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.994s user 0.001s sys 0.248s
I20260808 04:46:44.426843 4964 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 04:46:44.426970 4964 mini_ranger_kms.cc:401] 127.4.217.20:45599/kms/v1/keys
Aug 08 04:46:44 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164404, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.217.20@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
Aug 08 04:46:45 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164404, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.217.20@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
Aug 08 04:46:47 dist-test-slave-2kcd krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:46:47.291195 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:40803
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:40803
--ranger_config_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:45599/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:46:47.407763 23070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:47.408038 23070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:47.408123 23070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:47.412115 23070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:46:47.412196 23070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:47.412237 23070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:47.412294 23070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:46:47.412333 23070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:46:47.417682 23070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.217.20:45599/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:40803
--ranger_config_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:40803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:47.418649 23070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:47.419754 23070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:47.428660 23079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:47.428644 23076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:47.428644 23075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:47.429170 23070 server_base.cc:1034] running on GCE node
I20260808 04:46:47.429783 23070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:47.430835 23070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:47.432058 23070 hybrid_clock.cc:648] HybridClock initialized: now 1786164407432032 us; error 44 us; skew 500 ppm
Aug 08 04:46:47 dist-test-slave-2kcd krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:46:47.439875 23070 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:46:47.440991 23070 webserver.cc:460] Webserver started at http://127.4.217.62:41409/ using document root <none> and password file <none>
I20260808 04:46:47.441560 23070 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:47.441645 23070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:47.441866 23070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:46:47 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:46:47.498571 23070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "ee53d3a326be46c18846a9e34a8ae6dd"
format_stamp: "Formatted at 2026-08-08 04:46:47 on dist-test-slave-2kcd"
server_key: "77aa6e5761153ae2c310d4602b501a76"
server_key_iv: "83b2998c8e127481fba0e07aeaf3894e"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:47.499357 23070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "ee53d3a326be46c18846a9e34a8ae6dd"
format_stamp: "Formatted at 2026-08-08 04:46:47 on dist-test-slave-2kcd"
server_key: "77aa6e5761153ae2c310d4602b501a76"
server_key_iv: "83b2998c8e127481fba0e07aeaf3894e"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:47.503808 23070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:46:47.521934 23090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:47.523625 23070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:46:47.523772 23070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "ee53d3a326be46c18846a9e34a8ae6dd"
format_stamp: "Formatted at 2026-08-08 04:46:47 on dist-test-slave-2kcd"
server_key: "77aa6e5761153ae2c310d4602b501a76"
server_key_iv: "83b2998c8e127481fba0e07aeaf3894e"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:47.523921 23070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:47.543627 23070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:47.547060 23070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:47.547279 23070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:47.556851 23070 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:40803
I20260808 04:46:47.556910 23170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:40803 every 8 connection(s)
I20260808 04:46:47.558221 23070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:46:47.559372 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 23070
I20260808 04:46:47.559499 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:46:47.559831 4964 external_mini_cluster.cc:1428] Setting key 5d80447d4b3f10c8e93afe4a017a305c
I20260808 04:46:47.562016 23171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:46:47.568466 23171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd: Bootstrap starting.
Aug 08 04:46:47 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:46:47.571471 23171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd: Neither blocks nor log segments found. Creating new log.
I20260808 04:46:47.572359 23171 log.cc:824] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:46:47.574469 23171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd: No bootstrap required, opened a new log
I20260808 04:46:47.578118 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:47.561414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.4.217.62:40803)
0808 04:46:47.561937 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:47.561956 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:47.562000 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:46:47.562885 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:47.562910 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:47.563374 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:47.563743 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:47.564666 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:47.566299 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:47.567542 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:47.568302 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:47.568477 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:47.571818 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:47.571825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:47.571843 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:47.571876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:47.574643 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:47.575183 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:47.575190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:47.575196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:47.575332 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:47.575653 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:47.575656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:47.575658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:47.576172 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:47.576386 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:47.576689 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 04:46:47.577179 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":164,"threads_started":1}
I20260808 04:46:47.578012 23171 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee53d3a326be46c18846a9e34a8ae6dd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40803 } }
I20260808 04:46:47.578250 23171 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:46:47.578326 23171 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee53d3a326be46c18846a9e34a8ae6dd, State: Initialized, Role: FOLLOWER
I20260808 04:46:47.578770 23171 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee53d3a326be46c18846a9e34a8ae6dd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40803 } }
I20260808 04:46:47.578931 23171 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:46:47.579036 23171 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:46:47.579166 23171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:46:47.580056 23171 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee53d3a326be46c18846a9e34a8ae6dd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40803 } }
I20260808 04:46:47.580400 23171 leader_election.cc:302] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee53d3a326be46c18846a9e34a8ae6dd; no voters:
I20260808 04:46:47.580761 23171 leader_election.cc:288] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:46:47.580885 23176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:46:47.581164 23176 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [term 1 LEADER]: Becoming Leader. State: Replica: ee53d3a326be46c18846a9e34a8ae6dd, State: Running, Role: LEADER
I20260808 04:46:47.581512 23176 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee53d3a326be46c18846a9e34a8ae6dd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40803 } }
I20260808 04:46:47.582140 23171 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:46:47.582679 23176 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee53d3a326be46c18846a9e34a8ae6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee53d3a326be46c18846a9e34a8ae6dd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40803 } } }
I20260808 04:46:47.582875 23176 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [sys.catalog]: This master's current role is: LEADER
I20260808 04:46:47.583133 23177 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee53d3a326be46c18846a9e34a8ae6dd. Latest consensus state: current_term: 1 leader_uuid: "ee53d3a326be46c18846a9e34a8ae6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee53d3a326be46c18846a9e34a8ae6dd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40803 } } }
I20260808 04:46:47.583302 23177 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd [sys.catalog]: This master's current role is: LEADER
I20260808 04:46:47.583346 23184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:46:47.586802 23184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:46:47.587282 23171 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:46:47.597958 23184 catalog_manager.cc:1347] Generated new cluster ID: 276fe31ffcd347d5957c7b9b070148fa
I20260808 04:46:47.598316 23184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:46:47.613305 23184 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:46:47.614601 23184 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:46:47.637292 23184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd: Generated new TSK 0
I20260808 04:46:47.638235 23184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:46:48.300074 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:45599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:40803
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:46:48.455293 23237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:48.455619 23237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:48.455791 23237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:48.461230 23237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:48.461391 23237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:48.461536 23237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:46:48.468081 23237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.217.20:45599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:40803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:48.469220 23237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:48.470235 23237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:48.482127 23255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:48.482153 23248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:48.482336 23251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:48.482803 23237 server_base.cc:1034] running on GCE node
I20260808 04:46:48.483178 23237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:48.483812 23237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:48.485011 23237 hybrid_clock.cc:648] HybridClock initialized: now 1786164408484981 us; error 44 us; skew 500 ppm
Aug 08 04:46:48 dist-test-slave-2kcd krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164408, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:46:48.488540 23237 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:46:48.489630 23237 webserver.cc:460] Webserver started at http://127.4.217.1:33915/ using document root <none> and password file <none>
I20260808 04:46:48.490376 23237 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:48.490509 23237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:48.490793 23237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:46:48 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164408, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:46:48.510933 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "09c8fa5553674b169b5499319256c1fd"
format_stamp: "Formatted at 2026-08-08 04:46:48 on dist-test-slave-2kcd"
server_key: "8893b966394382653ac907f193d76f12"
server_key_iv: "701c3d0df69bc66fa02ee769753c7be2"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:48.511775 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "09c8fa5553674b169b5499319256c1fd"
format_stamp: "Formatted at 2026-08-08 04:46:48 on dist-test-slave-2kcd"
server_key: "8893b966394382653ac907f193d76f12"
server_key_iv: "701c3d0df69bc66fa02ee769753c7be2"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:48.517165 23237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:46:48.551718 23270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:48.565984 23237 fs_manager.cc:725] Time spent opening block manager: real 0.033s user 0.001s sys 0.000s
I20260808 04:46:48.566241 23237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "09c8fa5553674b169b5499319256c1fd"
format_stamp: "Formatted at 2026-08-08 04:46:48 on dist-test-slave-2kcd"
server_key: "8893b966394382653ac907f193d76f12"
server_key_iv: "701c3d0df69bc66fa02ee769753c7be2"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:48.566474 23237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:48.587675 23237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:48.591323 23237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:48.591562 23237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:48.592391 23237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:46:48.593448 23237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:46:48.593545 23237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:48.593631 23237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:46:48.593696 23237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:48.606153 23237 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:42469
I20260808 04:46:48.606242 23444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:42469 every 8 connection(s)
I20260808 04:46:48.608557 23237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:46:48.614560 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 23237
I20260808 04:46:48.614691 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:46:48.615024 4964 external_mini_cluster.cc:1428] Setting key a2b9934c1369a84f10e32ddbb9fd4538
Aug 08 04:46:48 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164408, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
I20260808 04:46:48.629109 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:48.612158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:55095 (local address 127.4.217.62:40803)
0808 04:46:48.612431 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:48.612441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:48.613397 (+ 956us) server_negotiation.cc:407] Connection header received
0808 04:46:48.614699 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:48.614703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:48.614764 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:48.614829 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:48.617143 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:48.617995 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:48.619736 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:48.620126 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:48.620258 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:48.623455 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:48.623459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:48.623462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:48.623492 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:48.625518 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:48.626209 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:48.626213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:48.626215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:48.626314 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:48.626741 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:48.626745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:48.626747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:48.626998 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:48.627101 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:48.628713 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0808 04:46:48.628894 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":108,"threads_started":1}
I20260808 04:46:48.630082 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:48.612652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:40803 (local address 127.4.217.1:55095)
0808 04:46:48.613238 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:48.613293 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:48.614438 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:48.615284 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:48.615292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:48.615670 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:46:48.616954 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:48.616974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:48.618171 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:48.618174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:48.619593 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:48.619605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:48.620460 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:48.620464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:48.620666 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:48.621389 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:46:48.621411 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:46:48.623236 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:46:48.625669 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:48.625677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:48.625696 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:48.626083 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:48.626423 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:48.626426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:48.626428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:48.626620 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:48.627166 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:46:48.627174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:46:48.627530 (+ 356us) client_negotiation.cc:769] Sending connection context
0808 04:46:48.627768 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 04:46:48.628131 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:46:48.631700 23445 heartbeater.cc:347] Connected to a master server at 127.4.217.62:40803
I20260808 04:46:48.631949 23445 heartbeater.cc:464] Registering TS with master...
I20260808 04:46:48.632514 23445 heartbeater.cc:511] Master 127.4.217.62:40803 requested a full tablet report, sending...
I20260808 04:46:48.634313 23114 ts_manager.cc:194] Registered new tserver with Master: 09c8fa5553674b169b5499319256c1fd (127.4.217.1:42469)
I20260808 04:46:48.635818 23114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:55095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:46:48.654836 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:45599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:40803
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:46:48 dist-test-slave-2kcd krb5kdc[22335](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164408, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 04:46:48.817083 23455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:48.817422 23455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:48.817485 23455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:48.823459 23455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:48.823550 23455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:48.823640 23455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:46:48.831416 23455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.217.20:45599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:40803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:48.832832 23455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:48.834175 23455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:48.847294 23471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:48.848146 23465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:48.848434 23468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:48.849133 23455 server_base.cc:1034] running on GCE node
I20260808 04:46:48.849905 23455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:48.850754 23455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:48.851917 23455 hybrid_clock.cc:648] HybridClock initialized: now 1786164408851878 us; error 54 us; skew 500 ppm
Aug 08 04:46:48 dist-test-slave-2kcd krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164408, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:46:48.856137 23455 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:46:48.857635 23455 webserver.cc:460] Webserver started at http://127.4.217.2:39071/ using document root <none> and password file <none>
I20260808 04:46:48.858433 23455 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:48.858556 23455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:48.858831 23455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:46:48 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164408, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:46:48.876590 23455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "cc5ecbccf4964e50b6880aaf71ffa35d"
format_stamp: "Formatted at 2026-08-08 04:46:48 on dist-test-slave-2kcd"
server_key: "5cc48310864bb1b2e8790c31339b5aa6"
server_key_iv: "53482378003a56307ddbc0203df17ad2"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:48.877514 23455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "cc5ecbccf4964e50b6880aaf71ffa35d"
format_stamp: "Formatted at 2026-08-08 04:46:48 on dist-test-slave-2kcd"
server_key: "5cc48310864bb1b2e8790c31339b5aa6"
server_key_iv: "53482378003a56307ddbc0203df17ad2"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:48.894246 23455 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s
I20260808 04:46:48.911435 23482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:48.913160 23455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:46:48.913391 23455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "cc5ecbccf4964e50b6880aaf71ffa35d"
format_stamp: "Formatted at 2026-08-08 04:46:48 on dist-test-slave-2kcd"
server_key: "5cc48310864bb1b2e8790c31339b5aa6"
server_key_iv: "53482378003a56307ddbc0203df17ad2"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:48.913620 23455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:48.939806 23455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:48.943576 23455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:48.943783 23455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:48.944516 23455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:46:48.945703 23455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:46:48.945770 23455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:48.945854 23455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:46:48.945889 23455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:48.957494 23455 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:41357
I20260808 04:46:48.958681 23455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:46:48.965348 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 23455
I20260808 04:46:48.965480 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:46:48.965806 4964 external_mini_cluster.cc:1428] Setting key 76eea93aac619b98c253261b19b1708c
I20260808 04:46:48.974360 23686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:41357 every 8 connection(s)
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
Aug 08 04:46:48 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164408, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
I20260808 04:46:49.015393 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:48.969849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:41655 (local address 127.4.217.62:40803)
0808 04:46:48.970007 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:48.970015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:48.977341 (+ 7326us) server_negotiation.cc:407] Connection header received
0808 04:46:48.978714 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:48.978718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:48.978777 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:48.978874 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:48.984330 (+ 5456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:48.985226 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:48.988958 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:48.989413 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:48.989506 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.007931 (+ 18425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:49.007937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:49.007940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:49.007982 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:49.010005 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:49.011662 (+ 1657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:49.011666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:49.011668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:49.011764 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:49.012248 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:49.012251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:49.012253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:49.012526 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:49.012620 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:49.015042 (+ 2422us) server_negotiation.cc:299] Negotiation successful
0808 04:46:49.015180 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:46:49.016482 23691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:48.968317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:40803 (local address 127.4.217.2:41655)
0808 04:46:48.977147 (+ 8830us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:48.977216 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:48.978438 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:48.979209 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:48.979220 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:48.980676 (+ 1456us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:46:48.981525 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:48.981547 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:48.985595 (+ 4048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:48.985599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:48.988759 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:48.988796 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:48.992666 (+ 3870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:48.992673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:48.992933 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:48.993706 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:46:48.993731 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:46:49.007613 (+ 13882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:46:49.010182 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:49.010189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:49.010210 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:49.010646 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:49.011891 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:49.011894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:49.011896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:49.012120 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:49.012698 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:46:49.012706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:46:49.013723 (+ 1017us) client_negotiation.cc:769] Sending connection context
0808 04:46:49.015015 (+ 1292us) client_negotiation.cc:240] Negotiation successful
0808 04:46:49.015376 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8527,"thread_start_us":179,"threads_started":1}
I20260808 04:46:49.017889 23687 heartbeater.cc:347] Connected to a master server at 127.4.217.62:40803
I20260808 04:46:49.018151 23687 heartbeater.cc:464] Registering TS with master...
I20260808 04:46:49.018781 23687 heartbeater.cc:511] Master 127.4.217.62:40803 requested a full tablet report, sending...
I20260808 04:46:49.020056 23114 ts_manager.cc:194] Registered new tserver with Master: cc5ecbccf4964e50b6880aaf71ffa35d (127.4.217.2:41357)
I20260808 04:46:49.020908 23114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:41655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:46:49.050014 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:45599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:40803
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:46:49 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164408, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
W20260808 04:46:49.246833 23702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:46:49.247293 23702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:46:49.247450 23702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:46:49.253926 23702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:46:49.254057 23702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:46:49.254184 23702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:46:49.261629 23702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.217.20:45599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:40803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:46:49.263340 23702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:46:49.264806 23702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:46:49.278227 23713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:49.279327 23702 server_base.cc:1034] running on GCE node
W20260808 04:46:49.281262 23711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:46:49.281366 23710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:46:49.282064 23702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:46:49.282935 23702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:46:49.286227 23702 hybrid_clock.cc:648] HybridClock initialized: now 1786164409286149 us; error 87 us; skew 500 ppm
Aug 08 04:46:49 dist-test-slave-2kcd krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164409, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:46:49.290066 23702 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:46:49.291229 23702 webserver.cc:460] Webserver started at http://127.4.217.3:45245/ using document root <none> and password file <none>
I20260808 04:46:49.292088 23702 fs_manager.cc:359] Metadata directory not provided
I20260808 04:46:49.292171 23702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:46:49.292402 23702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:46:49 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164409, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:46:49.309132 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "f57f2b827b3e4c85b45e238701338847"
format_stamp: "Formatted at 2026-08-08 04:46:49 on dist-test-slave-2kcd"
server_key: "43ca8c26d84565b6996d9e11d33c6935"
server_key_iv: "0445c864d18a5dfc6be309d035003085"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:49.310621 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "f57f2b827b3e4c85b45e238701338847"
format_stamp: "Formatted at 2026-08-08 04:46:49 on dist-test-slave-2kcd"
server_key: "43ca8c26d84565b6996d9e11d33c6935"
server_key_iv: "0445c864d18a5dfc6be309d035003085"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:49.315871 23702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:46:49.340145 23724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:49.341889 23702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 04:46:49.342023 23702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "f57f2b827b3e4c85b45e238701338847"
format_stamp: "Formatted at 2026-08-08 04:46:49 on dist-test-slave-2kcd"
server_key: "43ca8c26d84565b6996d9e11d33c6935"
server_key_iv: "0445c864d18a5dfc6be309d035003085"
server_key_version: "kuduclusterkey@0"
I20260808 04:46:49.342154 23702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:46:49.372257 23702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:46:49.376003 23702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:46:49.376252 23702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:46:49.376962 23702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:46:49.378059 23702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:46:49.378166 23702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:49.378263 23702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:46:49.378325 23702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:46:49.398713 23702 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:35157
I20260808 04:46:49.399982 23702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:46:49.401531 23908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:35157 every 8 connection(s)
I20260808 04:46:49.405359 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 23702
I20260808 04:46:49.405489 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:46:49.405808 4964 external_mini_cluster.cc:1428] Setting key 69e0a60cf26f4f9cb347b43bf916431f
Aug 08 04:46:49 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164409, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:46:49.422196 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.403571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:44909 (local address 127.4.217.62:40803)
0808 04:46:49.403716 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:49.403721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:49.404920 (+ 1199us) server_negotiation.cc:407] Connection header received
0808 04:46:49.406909 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:49.406913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:49.406971 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:49.407027 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:49.409909 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.410760 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.412656 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.413041 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.413122 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.416319 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:49.416323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:49.416326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:49.416354 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:49.418186 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:49.420165 (+ 1979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:49.420169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:49.420171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:49.420286 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:49.420763 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:49.420767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:49.420769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:49.421026 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:49.421103 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:49.421899 (+ 796us) server_negotiation.cc:299] Negotiation successful
0808 04:46:49.422025 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:46:49.424132 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.404046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:40803 (local address 127.4.217.3:44909)
0808 04:46:49.404709 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:49.404771 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:49.405943 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:49.408504 (+ 2561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:49.408514 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:49.409048 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:46:49.409718 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.409736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.411111 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.411114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.412515 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.412527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.413336 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.413340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.413532 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.414293 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:46:49.414315 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:46:49.416114 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:46:49.418333 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:49.418338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:49.418354 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:49.420045 (+ 1691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:49.420460 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:46:49.420463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:46:49.420465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:46:49.420643 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:46:49.421192 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:46:49.421199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:46:49.421595 (+ 396us) client_negotiation.cc:769] Sending connection context
0808 04:46:49.422648 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0808 04:46:49.423000 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":200,"threads_started":1}
I20260808 04:46:49.425869 23909 heartbeater.cc:347] Connected to a master server at 127.4.217.62:40803
I20260808 04:46:49.426132 23909 heartbeater.cc:464] Registering TS with master...
I20260808 04:46:49.426752 23909 heartbeater.cc:511] Master 127.4.217.62:40803 requested a full tablet report, sending...
I20260808 04:46:49.427992 23114 ts_manager.cc:194] Registered new tserver with Master: f57f2b827b3e4c85b45e238701338847 (127.4.217.3:35157)
I20260808 04:46:49.428893 23114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:44909
I20260808 04:46:49.433351 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:46:49.447561 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.437058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.4.217.62:40803)
0808 04:46:49.437173 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:49.437177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:49.437818 (+ 641us) server_negotiation.cc:407] Connection header received
0808 04:46:49.438007 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:49.438010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:49.438060 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:49.438111 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:46:49.439117 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.439940 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.441237 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.441594 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.441667 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.443573 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:46:49.443577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:46:49.443580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:46:49.443608 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:46:49.445461 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:49.445995 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:49.445999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:49.446001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:49.446094 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:46:49.446501 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:46:49.446505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:46:49.446507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:46:49.446730 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:46:49.446843 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:49.447236 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 04:46:49.447384 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:46:49.451579 23114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:46:49.456362 23114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:46:49.480146 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.471092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:35157 (local address 127.0.0.1:60496)
0808 04:46:49.472590 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:49.472604 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:49.472710 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:49.473409 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:49.473417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:49.473463 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:49.473645 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.473655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.477009 (+ 3354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.477013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.478111 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.478123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.479531 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.479536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.479658 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.479702 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 04:46:49.479819 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 04:46:49.479972 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1433,"thread_start_us":82,"threads_started":1}
I20260808 04:46:49.480774 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.471255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.4.217.3:35157)
0808 04:46:49.472438 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:49.472452 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:49.472837 (+ 385us) server_negotiation.cc:407] Connection header received
0808 04:46:49.472966 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:49.472973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:49.473190 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:49.473360 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:49.473805 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.475759 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.478288 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.479153 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.479269 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.479442 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:49.480510 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0808 04:46:49.480620 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":161,"threads_started":1}
I20260808 04:46:49.481729 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.472062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:41357 (local address 127.0.0.1:46376)
0808 04:46:49.473818 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:49.473832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:49.473948 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:49.474647 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:49.474650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:49.474669 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:49.474798 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.474807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.476244 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.476247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.477661 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.477673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.481308 (+ 3635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.481312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.481419 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.481444 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:46:49.481522 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:46:49.481594 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1695,"thread_start_us":92,"threads_started":1}
I20260808 04:46:49.484719 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.472222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.4.217.2:41357)
0808 04:46:49.473665 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:49.473686 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:49.474055 (+ 369us) server_negotiation.cc:407] Connection header received
0808 04:46:49.474145 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:49.474150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:49.474337 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:49.474495 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:49.474975 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.476102 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.480138 (+ 4036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.480973 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.481094 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.481239 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:49.484467 (+ 3228us) server_negotiation.cc:299] Negotiation successful
0808 04:46:49.484570 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":155,"threads_started":1}
I20260808 04:46:49.485363 23798 tablet_service.cc:1467] Processing CreateTablet for tablet 262d3a16bce7451cb011c7932060afb8 (DEFAULT_TABLE table=test-table [id=2c7e089800f34ca9a200b84c74c1ddcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:46:49.486189 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.470137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:42469 (local address 127.0.0.1:42636)
0808 04:46:49.471897 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:49.471961 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:49.472096 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:49.478430 (+ 6334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:49.478434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:49.478453 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:49.478570 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.478576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.482108 (+ 3532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.482111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.483249 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.483261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.485784 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.485788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.485896 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.485934 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:46:49.485996 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 04:46:49.486059 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1580,"thread_start_us":108,"threads_started":1}
I20260808 04:46:49.486613 23798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262d3a16bce7451cb011c7932060afb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:46:49.486971 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.470368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.4.217.1:42469)
0808 04:46:49.477971 (+ 7603us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:49.477979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:49.478003 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:46:49.478075 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:49.478080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:49.478268 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:49.478727 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:49.478878 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.479980 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.483415 (+ 3435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.484322 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.486530 (+ 2208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.486675 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:49.486728 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 04:46:49.486817 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7475,"thread_start_us":71,"threads_started":1}
I20260808 04:46:49.487632 23557 tablet_service.cc:1467] Processing CreateTablet for tablet 262d3a16bce7451cb011c7932060afb8 (DEFAULT_TABLE table=test-table [id=2c7e089800f34ca9a200b84c74c1ddcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:46:49.488643 23557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262d3a16bce7451cb011c7932060afb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:46:49.495973 23935 tablet_bootstrap.cc:492] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: Bootstrap starting.
I20260808 04:46:49.496018 23349 tablet_service.cc:1467] Processing CreateTablet for tablet 262d3a16bce7451cb011c7932060afb8 (DEFAULT_TABLE table=test-table [id=2c7e089800f34ca9a200b84c74c1ddcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:46:49.497040 23349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262d3a16bce7451cb011c7932060afb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:46:49.498852 23935 tablet_bootstrap.cc:654] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: Neither blocks nor log segments found. Creating new log.
I20260808 04:46:49.499712 23935 log.cc:824] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: Log is configured to *not* fsync() on all Append() calls
I20260808 04:46:49.502705 23935 tablet_bootstrap.cc:492] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: No bootstrap required, opened a new log
I20260808 04:46:49.503011 23935 ts_tablet_manager.cc:1397] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:46:49.505651 23939 tablet_bootstrap.cc:492] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: Bootstrap starting.
I20260808 04:46:49.508666 23939 tablet_bootstrap.cc:654] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: Neither blocks nor log segments found. Creating new log.
I20260808 04:46:49.509089 23942 tablet_bootstrap.cc:492] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd: Bootstrap starting.
I20260808 04:46:49.511400 23942 tablet_bootstrap.cc:654] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd: Neither blocks nor log segments found. Creating new log.
I20260808 04:46:49.512379 23939 log.cc:824] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:46:49.513772 23942 log.cc:824] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:46:49.516003 23939 tablet_bootstrap.cc:492] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: No bootstrap required, opened a new log
I20260808 04:46:49.516261 23939 ts_tablet_manager.cc:1397] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260808 04:46:49.520696 23935 raft_consensus.cc:348] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } }
I20260808 04:46:49.521287 23935 raft_consensus.cc:374] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:46:49.521451 23935 raft_consensus.cc:729] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57f2b827b3e4c85b45e238701338847, State: Initialized, Role: FOLLOWER
I20260808 04:46:49.521533 23942 tablet_bootstrap.cc:492] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd: No bootstrap required, opened a new log
I20260808 04:46:49.521777 23942 ts_tablet_manager.cc:1397] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20260808 04:46:49.522112 23935 consensus_queue.cc:260] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } }
I20260808 04:46:49.523322 23935 ts_tablet_manager.cc:1428] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260808 04:46:49.524681 23909 heartbeater.cc:503] Master 127.4.217.62:40803 was elected leader, sending a full tablet report...
I20260808 04:46:49.526381 23939 raft_consensus.cc:348] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } }
I20260808 04:46:49.526716 23939 raft_consensus.cc:374] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:46:49.526824 23939 raft_consensus.cc:729] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5ecbccf4964e50b6880aaf71ffa35d, State: Initialized, Role: FOLLOWER
I20260808 04:46:49.527377 23939 consensus_queue.cc:260] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } }
I20260808 04:46:49.528489 23939 ts_tablet_manager.cc:1428] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: Time spent starting tablet: real 0.012s user 0.001s sys 0.004s
I20260808 04:46:49.528924 23687 heartbeater.cc:503] Master 127.4.217.62:40803 was elected leader, sending a full tablet report...
Aug 08 04:46:49 dist-test-slave-2kcd krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164408, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:46:49.530797 23942 raft_consensus.cc:348] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } }
I20260808 04:46:49.531128 23942 raft_consensus.cc:374] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:46:49.531242 23942 raft_consensus.cc:729] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c8fa5553674b169b5499319256c1fd, State: Initialized, Role: FOLLOWER
I20260808 04:46:49.532673 23942 consensus_queue.cc:260] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } }
I20260808 04:46:49.533962 23942 ts_tablet_manager.cc:1428] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260808 04:46:49.534981 23445 heartbeater.cc:503] Master 127.4.217.62:40803 was elected leader, sending a full tablet report...
W20260808 04:46:49.612288 23446 tablet.cc:2369] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:46:49.637182 23948 raft_consensus.cc:482] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:46:49.637424 23948 raft_consensus.cc:504] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } }
I20260808 04:46:49.638911 23948 leader_election.cc:288] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f57f2b827b3e4c85b45e238701338847 (127.4.217.3:35157), cc5ecbccf4964e50b6880aaf71ffa35d (127.4.217.2:41357)
W20260808 04:46:49.651611 23910 tablet.cc:2369] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:46:49.652606 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.639908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:59247 (local address 127.4.217.2:41357)
0808 04:46:49.640025 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:49.640029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:49.648333 (+ 8304us) server_negotiation.cc:407] Connection header received
0808 04:46:49.648531 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:49.648535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:49.648605 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:49.648650 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:49.649068 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.649825 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.651248 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.651857 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.651944 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.652009 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:49.652378 (+ 369us) server_negotiation.cc:299] Negotiation successful
0808 04:46:49.652464 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 04:46:49.653043 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.639314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:41357 (local address 127.4.217.1:59247)
0808 04:46:49.648231 (+ 8917us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:49.648248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:49.648415 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:49.648796 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:49.648799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:49.648818 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:49.648925 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.648934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.649973 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.649978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.651086 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.651101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.652119 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.652123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.652208 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.652234 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:46:49.652769 (+ 535us) client_negotiation.cc:240] Negotiation successful
0808 04:46:49.652873 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8838,"thread_start_us":112,"threads_started":1}
I20260808 04:46:49.653301 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.639594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:40697 (local address 127.4.217.3:35157)
0808 04:46:49.639725 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:49.639728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:49.644971 (+ 5243us) server_negotiation.cc:407] Connection header received
0808 04:46:49.647401 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:49.647406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:49.647481 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:49.647538 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:46:49.648022 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.648803 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.650544 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.651168 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.651279 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.651354 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:49.652661 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0808 04:46:49.652741 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:46:49.653946 23598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262d3a16bce7451cb011c7932060afb8" candidate_uuid: "09c8fa5553674b169b5499319256c1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" is_pre_election: true
I20260808 04:46:49.654301 23598 raft_consensus.cc:2454] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c8fa5553674b169b5499319256c1fd in term 0.
I20260808 04:46:49.655053 23276 leader_election.cc:302] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c8fa5553674b169b5499319256c1fd, cc5ecbccf4964e50b6880aaf71ffa35d; no voters:
I20260808 04:46:49.655396 23948 raft_consensus.cc:2781] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:46:49.655474 23948 raft_consensus.cc:482] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:46:49.655514 23948 raft_consensus.cc:3037] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:46:49.656945 23948 raft_consensus.cc:504] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } }
I20260808 04:46:49.657359 23948 leader_election.cc:288] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [CANDIDATE]: Term 1 election: Requested vote from peers f57f2b827b3e4c85b45e238701338847 (127.4.217.3:35157), cc5ecbccf4964e50b6880aaf71ffa35d (127.4.217.2:41357)
I20260808 04:46:49.658008 23598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262d3a16bce7451cb011c7932060afb8" candidate_uuid: "09c8fa5553674b169b5499319256c1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d"
I20260808 04:46:49.658141 23598 raft_consensus.cc:3037] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:46:49.659514 23598 raft_consensus.cc:2454] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c8fa5553674b169b5499319256c1fd in term 1.
I20260808 04:46:49.659929 23276 leader_election.cc:302] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c8fa5553674b169b5499319256c1fd, cc5ecbccf4964e50b6880aaf71ffa35d; no voters:
I20260808 04:46:49.660192 23948 raft_consensus.cc:2781] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:46:49.660414 23948 raft_consensus.cc:686] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [term 1 LEADER]: Becoming Leader. State: Replica: 09c8fa5553674b169b5499319256c1fd, State: Running, Role: LEADER
I20260808 04:46:49.660863 23948 consensus_queue.cc:237] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } }
I20260808 04:46:49.664395 23114 catalog_manager.cc:5495] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 09c8fa5553674b169b5499319256c1fd (127.4.217.1). New cstate: current_term: 1 leader_uuid: "09c8fa5553674b169b5499319256c1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c8fa5553674b169b5499319256c1fd" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 } health_report { overall_health: UNKNOWN } } }
I20260808 04:46:49.665966 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.639144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:35157 (local address 127.4.217.1:40697)
0808 04:46:49.644829 (+ 5685us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:49.644858 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 04:46:49.647222 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:46:49.647733 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:46:49.647736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:46:49.647758 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:46:49.647883 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.647890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.649049 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.649052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.650396 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:46:49.650406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.652410 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.652414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:46:49.652506 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.652531 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:46:49.665704 (+ 13173us) client_negotiation.cc:240] Negotiation successful
0808 04:46:49.665812 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5561,"thread_start_us":123,"threads_started":1}
I20260808 04:46:49.667001 23834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262d3a16bce7451cb011c7932060afb8" candidate_uuid: "09c8fa5553674b169b5499319256c1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f57f2b827b3e4c85b45e238701338847" is_pre_election: true
I20260808 04:46:49.667450 23834 raft_consensus.cc:2454] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c8fa5553674b169b5499319256c1fd in term 0.
I20260808 04:46:49.667786 23833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262d3a16bce7451cb011c7932060afb8" candidate_uuid: "09c8fa5553674b169b5499319256c1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f57f2b827b3e4c85b45e238701338847"
I20260808 04:46:49.667883 23833 raft_consensus.cc:3037] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:46:49.669329 23833 raft_consensus.cc:2454] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c8fa5553674b169b5499319256c1fd in term 1.
I20260808 04:46:49.684335 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.678943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.4.217.1:42469)
0808 04:46:49.679129 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 04:46:49.679134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:46:49.679151 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:46:49.679229 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:46:49.679232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:46:49.679284 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:46:49.679410 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:46:49.679918 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.680699 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.682017 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:46:49.682394 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:46:49.682493 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:46:49.682952 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:46:49.683534 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:46:49.683652 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:46:49.684104 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 04:46:49.684168 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260808 04:46:49.691676 23598 raft_consensus.cc:1261] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [term 1 FOLLOWER]: Refusing update from remote peer 09c8fa5553674b169b5499319256c1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:46:49.692698 23833 raft_consensus.cc:1261] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [term 1 FOLLOWER]: Refusing update from remote peer 09c8fa5553674b169b5499319256c1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:46:49.692911 23960 consensus_queue.cc:1035] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5ecbccf4964e50b6880aaf71ffa35d" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:46:49.693472 23960 consensus_queue.cc:1035] T 262d3a16bce7451cb011c7932060afb8 P 09c8fa5553674b169b5499319256c1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f57f2b827b3e4c85b45e238701338847" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 35157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:46:49.707931 23966 mvcc.cc:205] Tried to move back new op lower bound from 7316129422087921664 to 7316129421974650880. Current Snapshot: MvccSnapshot[applied={T|T < 7316129422087921664}]
I20260808 04:46:49.709790 23968 mvcc.cc:205] Tried to move back new op lower bound from 7316129422087921664 to 7316129421974650880. Current Snapshot: MvccSnapshot[applied={T|T < 7316129422087921664}]
I20260808 04:46:49.717832 23967 mvcc.cc:205] Tried to move back new op lower bound from 7316129422087921664 to 7316129421974650880. Current Snapshot: MvccSnapshot[applied={T|T < 7316129422087921664}]
I20260808 04:46:49.728823 23114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:46:49.729031 23114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:46:49.731949 23114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee53d3a326be46c18846a9e34a8ae6dd: Sending DeleteTablet for 3 replicas of tablet 262d3a16bce7451cb011c7932060afb8
I20260808 04:46:49.733242 23349 tablet_service.cc:1514] Processing DeleteTablet for tablet 262d3a16bce7451cb011c7932060afb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:46:49 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:42636
I20260808 04:46:49.733318 23557 tablet_service.cc:1514] Processing DeleteTablet for tablet 262d3a16bce7451cb011c7932060afb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:46:49 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:46376
I20260808 04:46:49.734306 23798 tablet_service.cc:1514] Processing DeleteTablet for tablet 262d3a16bce7451cb011c7932060afb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:46:49 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:60496
I20260808 04:46:49.736955 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 23237
I20260808 04:46:49.759881 23985 tablet_replica.cc:330] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: stopping tablet replica
I20260808 04:46:49.760250 23986 tablet_replica.cc:330] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: stopping tablet replica
I20260808 04:46:49.760756 23986 raft_consensus.cc:2229] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:46:49.761199 23985 raft_consensus.cc:2229] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:46:49.761682 23986 raft_consensus.cc:2258] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:46:49.762055 23985 raft_consensus.cc:2258] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:46:49.763892 23985 ts_tablet_manager.cc:1905] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:46:49.765224 23986 ts_tablet_manager.cc:1905] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:46:49.767606 23986 ts_tablet_manager.cc:1918] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:46:49.767728 23986 log.cc:1196] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/262d3a16bce7451cb011c7932060afb8
I20260808 04:46:49.768309 23986 ts_tablet_manager.cc:1939] T 262d3a16bce7451cb011c7932060afb8 P f57f2b827b3e4c85b45e238701338847: Deleting consensus metadata
I20260808 04:46:49.770936 23985 ts_tablet_manager.cc:1918] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:46:49.771056 23985 log.cc:1196] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/262d3a16bce7451cb011c7932060afb8
I20260808 04:46:49.771375 23985 ts_tablet_manager.cc:1939] T 262d3a16bce7451cb011c7932060afb8 P cc5ecbccf4964e50b6880aaf71ffa35d: Deleting consensus metadata
I20260808 04:46:49.772953 23095 catalog_manager.cc:4841] TS cc5ecbccf4964e50b6880aaf71ffa35d (127.4.217.2:41357): tablet 262d3a16bce7451cb011c7932060afb8 (table test-table [id=2c7e089800f34ca9a200b84c74c1ddcd]) successfully deleted
I20260808 04:46:49.773644 23096 catalog_manager.cc:4841] TS f57f2b827b3e4c85b45e238701338847 (127.4.217.3:35157): tablet 262d3a16bce7451cb011c7932060afb8 (table test-table [id=2c7e089800f34ca9a200b84c74c1ddcd]) successfully deleted
W20260808 04:46:49.784065 23095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:42469 (error 108)
I20260808 04:46:49.785936 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 23455
I20260808 04:46:49.785975 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:46:49.785635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:42469 (local address 127.0.0.1:42648)
0808 04:46:49.785792 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 04:46:49.785877 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:46:49.797227 23095 catalog_manager.cc:4584] TS 09c8fa5553674b169b5499319256c1fd (127.4.217.1:42469): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 262d3a16bce7451cb011c7932060afb8: Network error: Client connection negotiation failed: client connection to 127.4.217.1:42469: connect: Connection refused (error 111)
I20260808 04:46:49.799765 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 23702
I20260808 04:46:49.838411 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 23070
I20260808 04:46:49.852576 4964 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 04:46:50.197182 4964 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 04:46:50.197351 4964 mini_ranger.cc:66] Stopping Ranger...
I20260808 04:46:50.552057 4964 mini_ranger.cc:68] Stopped Ranger
2026-08-08 04:46:50.552 UTC [22382] LOG: received smart shutdown request
2026-08-08 04:46:50.555 UTC [22382] LOG: background worker "logical replication launcher" (PID 22397) exited with exit code 1
2026-08-08 04:46:50.556 UTC [22389] LOG: shutting down
2026-08-08 04:46:50.703 UTC [22382] LOG: database system is shut down
2026-08-08T04:46:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47595 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:46:50 dist-test-slave-2kcd krb5kdc[23996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:46:50 dist-test-slave-2kcd krb5kdc[23996](info): set up 2 sockets
Aug 08 04:46:50 dist-test-slave-2kcd krb5kdc[23996](info): commencing operation
krb5kdc: starting...
W20260808 04:46:52.117059 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:46:52 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:46:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/rangeradmin_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/rangeradmin_127.4.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/rangerlookup_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/rangerlookup_127.4.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/HTTP_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/HTTP_127.4.217.20@KRBTEST.COM.keytab.
I20260808 04:46:52.301692 4964 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskywCe24/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/postgres -l logfile start
2026-08-08 04:46:54.764 UTC [24043] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 04:46:54.764 UTC [24043] LOG: listening on IPv4 address "127.4.217.20", port 37221
2026-08-08 04:46:54.767 UTC [24043] LOG: listening on Unix socket "/tmp/.s.PGSQL.37221"
2026-08-08 04:46:54.804 UTC [24047] LOG: database system was shut down at 2026-08-08 04:46:53 UTC
2026-08-08 04:46:54.808 UTC [24043] LOG: database system is ready to accept connections
I20260808 04:46:55.971882 4964 mini_postgres.cc:96] Postgres bound to 37221
2026-08-08 04:46:55.979 UTC [24061] FATAL: database "slave" does not exist
127.4.217.20:37221 - accepting connections
I20260808 04:46:55.980229 4964 mini_ranger.cc:161] Starting Ranger...
I20260808 04:46:55.995889 4964 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 04:46:56.434054 4964 mini_ranger.cc:87] Created ranger Postgres database
I20260808 04:46:56.434130 4964 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-admin
2026-08-08 04:46:56,733 [I] DB FLAVOR :POSTGRES
2026-08-08 04:46:56,733 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:46:56,734 [I] Checking connection..
2026-08-08 04:46:56,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 04:46:57,001 [I] Checking connection passed.
2026-08-08 04:46:57,001 [I] --------- Verifying version history table ---------
2026-08-08 04:46:57,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:46:57,255 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:46:57,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:46:57,524 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:46:57,524 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 04:46:57,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 04:46:57.797 UTC [24159] WARNING: there is no transaction in progress
2026-08-08 04:46:57,806 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 04:46:57,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:46:58,080 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 04:46:58,080 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:46:58,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 04:46:58,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 04:46:58,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2kcd.c.gcp-upstream.internal','N') ;"
2026-08-08 04:46:58,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:46:59,085 [I] Table x_portal_user does not exist in database ranger
2026-08-08 04:46:59,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:46:59,330 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 04:46:59,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:46:59,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:46:59,855 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 04:46:59,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 04:47:00.809 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.824 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.839 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.855 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.871 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.888 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.938 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.944 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.958 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.967 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.978 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:00.991 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:01.001 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:01.007 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:01.014 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:01.020 UTC [24321] WARNING: there is no transaction in progress
2026-08-08 04:47:01,614 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 04:47:01,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 04:47:01,868 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 04:47:01,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:47:02,134 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 04:47:02,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:47:02,406 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 04:47:02,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:47:02,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:47:02,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2kcd.c.gcp-upstream.internal';"
2026-08-08 04:47:03,158 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 04:47:03,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 04:47:03,427 [I] DB_PATCHES have already been applied
I20260808 04:47:03.432879 4964 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-admin:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskywCe24/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 04:47:03.432943 4964 mini_ranger.cc:193] Using host: 127.4.217.20
I20260808 04:47:03.437425 4964 mini_ranger.cc:227] Ranger admin URL: http://127.4.217.20:55223
Aug 08, 2026 4:47:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:47:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:47:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 4:47:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 4:47:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 4:47:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 4:47:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 04:47:04.776073 4964 mini_ranger.cc:160] Time spent starting Ranger: real 8.796s user 0.000s sys 0.011s
Aug 08, 2026 4:47:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 4:47:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 08, 2026 4:47:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 4:47:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 4:47:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 04:47:25.600630 4964 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/rangerkms_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/rangerkms_127.4.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.217.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.217.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.217.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/HTTP_127.4.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.217.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/HTTP_127.4.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 04:47:25 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164445, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 04:47:25.686254 4964 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 04:47:25.696913 4964 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 04:47:26.485283 4964 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 04:47:26.485378 4964 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-kms
I20260808 04:47:26.485400 4964 mini_ranger_kms.cc:227] Using postgres at 127.4.217.20:37221
2026-08-08 04:47:26,673 [I] DB FLAVOR :POSTGRES
2026-08-08 04:47:26,674 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:47:26,674 [I] Checking connection
2026-08-08 04:47:26,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 04:47:26,909 [I] connection success
2026-08-08 04:47:26,909 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 04:47:26,909 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 04:47:26,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 04:47:27,181 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 04:47:27,181 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:47:27,181 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 04:47:27,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.217.20:37221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 04:47:27,451 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 04:47:27.891506 4964 mini_ranger_kms.cc:310] Created kms service
I20260808 04:47:28.110901 4964 mini_ranger_kms.cc:326] Created kudu user
I20260808 04:47:28.160116 4964 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 04:47:28 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164445, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:28.547465 4964 mini_ranger_kms.cc:384] Added ranger policy
I20260808 04:47:28.547629 4964 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-kms:/tmp/dist-test-taskywCe24/build/debug/bin/postgresql.jar:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskywCe24/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 04:47:28.547657 4964 mini_ranger_kms.cc:240] Using host: 127.4.217.20
I20260808 04:47:28.553318 4964 mini_ranger_kms.cc:276] Ranger KMS PID: 24625
I20260808 04:47:28.553434 4964 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.217.20:36121
Aug 08, 2026 4:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 4:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 4:47:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 4:47:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 04:47:30 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164450, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.217.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:31.317643 4964 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.631s user 0.027s sys 0.146s
I20260808 04:47:31.317791 4964 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 04:47:31.317870 4964 mini_ranger_kms.cc:401] 127.4.217.20:36121/kms/v1/keys
Aug 08 04:47:31 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164450, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.217.20@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
Aug 08 04:47:31 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164450, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.217.20@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
Aug 08 04:47:34 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:47:34.204142 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:37839
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:37839
--ranger_config_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:36121/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:34.321739 24713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:34.321978 24713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:34.322016 24713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:34.325932 24713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:47:34.325982 24713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:34.326001 24713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:34.326016 24713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:47:34.326031 24713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:47:34.331266 24713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.217.20:36121/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:37839
--ranger_config_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:37839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:34.332175 24713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:34.333196 24713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:34.341660 24724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:34.341820 24713 server_base.cc:1034] running on GCE node
W20260808 04:47:34.341671 24720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:34.341671 24718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:34.342527 24713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:34.343581 24713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:34.344771 24713 hybrid_clock.cc:648] HybridClock initialized: now 1786164454344756 us; error 31 us; skew 500 ppm
Aug 08 04:47:34 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164454, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:34.347492 24713 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:47:34.348374 24713 webserver.cc:460] Webserver started at http://127.4.217.62:45529/ using document root <none> and password file <none>
I20260808 04:47:34.348981 24713 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:34.349080 24713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:34.349314 24713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:47:34 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164454, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:34.403602 24713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "ca82db75b3a94ee4aad8cbe646e6f600"
format_stamp: "Formatted at 2026-08-08 04:47:34 on dist-test-slave-2kcd"
server_key: "cbcff30f46ac4a39dff14041661ded5d"
server_key_iv: "5c9fe31572d4377c3a1a74e9d25dc05b"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:34.404266 24713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "ca82db75b3a94ee4aad8cbe646e6f600"
format_stamp: "Formatted at 2026-08-08 04:47:34 on dist-test-slave-2kcd"
server_key: "cbcff30f46ac4a39dff14041661ded5d"
server_key_iv: "5c9fe31572d4377c3a1a74e9d25dc05b"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:34.408372 24713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:47:34.422516 24737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:34.424106 24713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:47:34.424356 24713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "ca82db75b3a94ee4aad8cbe646e6f600"
format_stamp: "Formatted at 2026-08-08 04:47:34 on dist-test-slave-2kcd"
server_key: "cbcff30f46ac4a39dff14041661ded5d"
server_key_iv: "5c9fe31572d4377c3a1a74e9d25dc05b"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:34.424659 24713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:34.442619 24713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:34.445722 24713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:34.445940 24713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:34.454469 24713 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:37839
I20260808 04:47:34.454474 24834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:37839 every 8 connection(s)
I20260808 04:47:34.455482 24713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:47:34.459170 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:34.461782 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 24713
I20260808 04:47:34.461932 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:47:34.462234 4964 external_mini_cluster.cc:1428] Setting key e1e5d9256c866013f5db6a6b4c37c777
I20260808 04:47:34.466995 24838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Bootstrap starting.
I20260808 04:47:34.469326 24838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:34.470206 24838 log.cc:824] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:47:34 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:34.472481 24838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: No bootstrap required, opened a new log
I20260808 04:47:34.476744 24838 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } }
I20260808 04:47:34.477031 24838 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:34.477089 24838 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca82db75b3a94ee4aad8cbe646e6f600, State: Initialized, Role: FOLLOWER
I20260808 04:47:34.477543 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:34.463704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.4.217.62:37839)
0808 04:47:34.464139 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:34.464155 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:34.464192 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:47:34.464930 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:34.464949 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:34.465284 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:34.465620 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:34.466470 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:34.467502 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:34.468377 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:34.468928 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:34.469120 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:34.471819 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:34.471826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:34.471843 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:34.471873 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:34.474546 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:34.474989 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:34.474995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:34.475001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:34.475141 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:34.475405 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:34.475409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:34.475410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:34.475906 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:34.476130 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:34.476318 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 04:47:34.476590 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":153,"threads_started":1}
I20260808 04:47:34.477581 24838 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } }
I20260808 04:47:34.477715 24838 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:47:34.477787 24838 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:47:34.477887 24838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:34.478955 24838 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } }
I20260808 04:47:34.479297 24838 leader_election.cc:302] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca82db75b3a94ee4aad8cbe646e6f600; no voters:
I20260808 04:47:34.479631 24838 leader_election.cc:288] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:47:34.479684 24844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:47:34.479957 24844 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 1 LEADER]: Becoming Leader. State: Replica: ca82db75b3a94ee4aad8cbe646e6f600, State: Running, Role: LEADER
I20260808 04:47:34.480321 24844 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } }
I20260808 04:47:34.480870 24838 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:47:34.481546 24844 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca82db75b3a94ee4aad8cbe646e6f600. Latest consensus state: current_term: 1 leader_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } } }
I20260808 04:47:34.481757 24844 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:34.482205 24859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:47:34.482184 24845 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } } }
I20260808 04:47:34.482436 24845 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:34.485288 24859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:47:34.485726 24838 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:47:34.494627 24859 catalog_manager.cc:1347] Generated new cluster ID: 5df821a992054349a4a34fe0f6108ce0
I20260808 04:47:34.494705 24859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:47:34.504021 24859 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:47:34.505338 24859 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:47:34.521296 24859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Generated new TSK 0
I20260808 04:47:34.522190 24859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:47:35.177284 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:36121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37839
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:35.329461 24914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:35.329823 24914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:35.329891 24914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:35.335969 24914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:35.336104 24914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:35.336234 24914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:47:35.344357 24914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.217.20:36121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:35.345788 24914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:35.347162 24914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:35.357237 24923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:35.357255 24921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:35.357234 24926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:35.357978 24914 server_base.cc:1034] running on GCE node
I20260808 04:47:35.358420 24914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:35.359138 24914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:35.360349 24914 hybrid_clock.cc:648] HybridClock initialized: now 1786164455360329 us; error 37 us; skew 500 ppm
Aug 08 04:47:35 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164455, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:35.363929 24914 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:47:35.365108 24914 webserver.cc:460] Webserver started at http://127.4.217.1:43827/ using document root <none> and password file <none>
I20260808 04:47:35.365801 24914 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:35.365911 24914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:35.366189 24914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:47:35 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164455, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:35.388015 24914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "5dc00a157fb74e989b8716d54614165c"
format_stamp: "Formatted at 2026-08-08 04:47:35 on dist-test-slave-2kcd"
server_key: "243bf774df3bb058cc488fcf612492b8"
server_key_iv: "24333784fe64fb3b9b35effa1c8e7058"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:35.388753 24914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "5dc00a157fb74e989b8716d54614165c"
format_stamp: "Formatted at 2026-08-08 04:47:35 on dist-test-slave-2kcd"
server_key: "243bf774df3bb058cc488fcf612492b8"
server_key_iv: "24333784fe64fb3b9b35effa1c8e7058"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:35.394182 24914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:47:35.408527 24943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:35.410259 24914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:47:35.410523 24914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "5dc00a157fb74e989b8716d54614165c"
format_stamp: "Formatted at 2026-08-08 04:47:35 on dist-test-slave-2kcd"
server_key: "243bf774df3bb058cc488fcf612492b8"
server_key_iv: "24333784fe64fb3b9b35effa1c8e7058"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:35.410749 24914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:35.445451 24914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:35.449396 24914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:35.449669 24914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:35.450462 24914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:35.451668 24914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:35.451773 24914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:35.451901 24914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:35.451987 24914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:35.463155 24914 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:37183
I20260808 04:47:35.464375 24914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:47:35.466921 25126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:37183 every 8 connection(s)
I20260808 04:47:35.469780 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 24914
I20260808 04:47:35.469905 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:47:35.470395 4964 external_mini_cluster.cc:1428] Setting key 0e11dd5ef5119a72e662a5e54b0eb892
Aug 08 04:47:35 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164455, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:47:35.491837 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:35.468972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:52089 (local address 127.4.217.62:37839)
0808 04:47:35.470546 (+ 1574us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:35.470555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:35.470572 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:47:35.471626 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:35.471630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:35.471690 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:35.471810 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:35.473270 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:35.473891 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:35.475376 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:35.475767 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:35.475856 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:35.484399 (+ 8543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:35.484403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:35.484406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:35.484438 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:35.486415 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:35.487134 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:35.487140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:35.487142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:35.487250 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:35.487710 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:35.487714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:35.487716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:35.488120 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:35.488214 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:35.491484 (+ 3270us) server_negotiation.cc:299] Negotiation successful
0808 04:47:35.491637 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1490,"thread_start_us":128,"threads_started":1}
I20260808 04:47:35.493878 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:35.469437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37839 (local address 127.4.217.1:52089)
0808 04:47:35.470345 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:35.470406 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:35.471410 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:35.472009 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:35.472018 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:35.472382 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:35.473118 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:35.473136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:35.474026 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:35.474030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:35.475215 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:35.475231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:35.476109 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:35.476113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:35.476339 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:35.477046 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:35.477068 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:35.484169 (+ 7101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:35.486562 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:35.486569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:35.486587 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:35.486950 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:35.487374 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:35.487377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:35.487380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:35.487579 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:35.488288 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:35.488295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:35.488733 (+ 438us) client_negotiation.cc:769] Sending connection context
0808 04:47:35.492319 (+ 3586us) client_negotiation.cc:240] Negotiation successful
0808 04:47:35.492709 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":153,"threads_started":1}
I20260808 04:47:35.495677 25129 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37839
I20260808 04:47:35.495932 25129 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:35.496572 25129 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
I20260808 04:47:35.498461 24777 ts_manager.cc:194] Registered new tserver with Master: 5dc00a157fb74e989b8716d54614165c (127.4.217.1:37183)
I20260808 04:47:35.500108 24777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:52089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:47:35.515453 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:36121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37839
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:47:35 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164455, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 04:47:35.645951 25142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:35.646219 25142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:35.646289 25142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:35.650650 25142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:35.650735 25142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:35.650835 25142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:47:35.656204 25142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.217.20:36121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:35.657292 25142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:35.658340 25142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:35.668978 25158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:35.669204 25162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:35.669461 25157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:35.669541 25142 server_base.cc:1034] running on GCE node
I20260808 04:47:35.670047 25142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:35.670738 25142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:35.672062 25142 hybrid_clock.cc:648] HybridClock initialized: now 1786164455672031 us; error 52 us; skew 500 ppm
Aug 08 04:47:35 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164455, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:35.674973 25142 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:47:35.675936 25142 webserver.cc:460] Webserver started at http://127.4.217.2:42335/ using document root <none> and password file <none>
I20260808 04:47:35.676537 25142 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:35.676631 25142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:35.676910 25142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:47:35 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164455, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:35.690132 25142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "20c76e9a35ba401384c7fe874387eaf4"
format_stamp: "Formatted at 2026-08-08 04:47:35 on dist-test-slave-2kcd"
server_key: "a7788f034d0851b63f8d11a961ddb997"
server_key_iv: "9b61356e74979a13074a6b2f50b2b232"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:35.690856 25142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "20c76e9a35ba401384c7fe874387eaf4"
format_stamp: "Formatted at 2026-08-08 04:47:35 on dist-test-slave-2kcd"
server_key: "a7788f034d0851b63f8d11a961ddb997"
server_key_iv: "9b61356e74979a13074a6b2f50b2b232"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:35.696497 25142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:47:35.716689 25174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:35.718587 25142 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260808 04:47:35.718812 25142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "20c76e9a35ba401384c7fe874387eaf4"
format_stamp: "Formatted at 2026-08-08 04:47:35 on dist-test-slave-2kcd"
server_key: "a7788f034d0851b63f8d11a961ddb997"
server_key_iv: "9b61356e74979a13074a6b2f50b2b232"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:35.719034 25142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:35.738950 25142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:35.744184 25142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:35.744459 25142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:35.745251 25142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:35.746345 25142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:35.746496 25142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:35.746608 25142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:35.746682 25142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:35.761123 25142 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:39751
I20260808 04:47:35.762437 25142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:47:35.764197 25355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:39751 every 8 connection(s)
I20260808 04:47:35.766686 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 25142
I20260808 04:47:35.766819 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:47:35.767158 4964 external_mini_cluster.cc:1428] Setting key 8d52a52967227b9c15a73b834bf793bd
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Aug 08 04:47:35 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164455, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:35.789330 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:35.766317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:38743 (local address 127.4.217.62:37839)
0808 04:47:35.766454 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:35.766458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:35.769175 (+ 2717us) server_negotiation.cc:407] Connection header received
0808 04:47:35.771604 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:35.771607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:35.771662 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:35.771758 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:35.778347 (+ 6589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:35.779202 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:35.780959 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:35.781315 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:35.781391 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:35.784895 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:35.784899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:35.784902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:35.784935 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:35.786700 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:35.787313 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:35.787317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:35.787319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:35.787403 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:35.787912 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:35.787915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:35.787917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:35.788144 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:35.788244 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:35.789011 (+ 767us) server_negotiation.cc:299] Negotiation successful
0808 04:47:35.789151 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:47:35.791196 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:35.766833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37839 (local address 127.4.217.2:38743)
0808 04:47:35.768934 (+ 2101us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:35.768990 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:35.770040 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:35.777073 (+ 7033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:35.777083 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:35.777450 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:35.778151 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:35.778172 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:35.779373 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:35.779376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:35.780816 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:35.780828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:35.781633 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:35.781637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:35.781954 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:35.782863 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:35.782885 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:35.784663 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:35.786841 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:35.786847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:35.786865 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:35.787196 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:35.787524 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:35.787527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:35.787529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:35.787792 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:35.788318 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:35.788325 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:35.788690 (+ 365us) client_negotiation.cc:769] Sending connection context
0808 04:47:35.789811 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0808 04:47:35.790158 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1858,"thread_start_us":160,"threads_started":1}
I20260808 04:47:35.792750 25357 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37839
I20260808 04:47:35.793016 25357 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:35.793627 25357 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
I20260808 04:47:35.794813 24777 ts_manager.cc:194] Registered new tserver with Master: 20c76e9a35ba401384c7fe874387eaf4 (127.4.217.2:39751)
I20260808 04:47:35.795581 24777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:38743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:47:35.815445 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:36121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37839
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:35.988831 25366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:35.989289 25366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:35.989431 25366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:35.995537 25366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:35.995692 25366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:35.995859 25366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:47:36.003950 25366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.217.20:36121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:36.005724 25366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:36.007237 25366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:36.026955 25375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:36.026970 25373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:36.028468 25366 server_base.cc:1034] running on GCE node
W20260808 04:47:36.028980 25371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:36.029578 25366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:36.030427 25366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:36.031661 25366 hybrid_clock.cc:648] HybridClock initialized: now 1786164456031588 us; error 85 us; skew 500 ppm
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:36.035499 25366 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:47:36.036563 25366 webserver.cc:460] Webserver started at http://127.4.217.3:41133/ using document root <none> and password file <none>
I20260808 04:47:36.037283 25366 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:36.037354 25366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:36.037585 25366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:36.065397 25366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "20b6101ceb884a328df7bca32799218e"
format_stamp: "Formatted at 2026-08-08 04:47:36 on dist-test-slave-2kcd"
server_key: "12c81c0c214b6250610d53e13c98646f"
server_key_iv: "24e7cc384b85a0b4631280feda7d1116"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:36.068212 25366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "20b6101ceb884a328df7bca32799218e"
format_stamp: "Formatted at 2026-08-08 04:47:36 on dist-test-slave-2kcd"
server_key: "12c81c0c214b6250610d53e13c98646f"
server_key_iv: "24e7cc384b85a0b4631280feda7d1116"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:36.073685 25366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:47:36.095834 25387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:36.097608 25366 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:47:36.097741 25366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "20b6101ceb884a328df7bca32799218e"
format_stamp: "Formatted at 2026-08-08 04:47:36 on dist-test-slave-2kcd"
server_key: "12c81c0c214b6250610d53e13c98646f"
server_key_iv: "24e7cc384b85a0b4631280feda7d1116"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:36.097868 25366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:36.125361 25366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:36.129379 25366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:36.129673 25366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:36.130465 25366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:36.131644 25366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:36.131758 25366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:36.131862 25366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:36.131959 25366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164455, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:36.144466 25366 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:39195
I20260808 04:47:36.145668 25366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:47:36.146916 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 25366
I20260808 04:47:36.147033 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:47:36.147369 4964 external_mini_cluster.cc:1428] Setting key 38e236260b61487a4b2779cb16b24e45
I20260808 04:47:36.148636 25572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:39195 every 8 connection(s)
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:36.166113 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:36.151961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:58807 (local address 127.4.217.62:37839)
0808 04:47:36.152080 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:36.152084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:36.152813 (+ 729us) server_negotiation.cc:407] Connection header received
0808 04:47:36.153967 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:36.153971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:36.154023 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:36.154071 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:36.155630 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:36.156371 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:36.157999 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:36.158326 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:36.158408 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:36.161539 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:36.161544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:36.161546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:36.161575 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:36.163346 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:36.164038 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:36.164042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:36.164044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:36.164204 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:36.164738 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:36.164742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:36.164744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:36.164996 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:36.165088 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:36.165805 (+ 717us) server_negotiation.cc:299] Negotiation successful
0808 04:47:36.165952 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:47:36.168093 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:36.150827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37839 (local address 127.4.217.3:58807)
0808 04:47:36.152656 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:36.152707 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:36.153734 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:36.154465 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:36.154485 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:36.154807 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:36.155460 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:36.155489 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:36.156547 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:36.156552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:36.157845 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:36.157859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:36.158623 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:36.158627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:36.158808 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:36.159531 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:36.159553 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:36.161326 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:36.163501 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:36.163507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:36.163522 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:36.163911 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:36.164332 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:36.164335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:36.164337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:36.164613 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:36.165161 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:36.165168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:36.165504 (+ 336us) client_negotiation.cc:769] Sending connection context
0808 04:47:36.166566 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0808 04:47:36.166905 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1602,"thread_start_us":243,"threads_started":1}
I20260808 04:47:36.169848 25573 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37839
I20260808 04:47:36.170112 25573 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:36.170691 25573 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
I20260808 04:47:36.171871 24777 ts_manager.cc:194] Registered new tserver with Master: 20b6101ceb884a328df7bca32799218e (127.4.217.3:39195)
I20260808 04:47:36.172636 24777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:58807
I20260808 04:47:36.174057 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:47:36.174906 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 24914
I20260808 04:47:36.191941 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 25142
I20260808 04:47:36.202651 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 25366
I20260808 04:47:36.218855 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 24713
I20260808 04:47:36.235236 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:37839
--webserver_interface=127.4.217.62
--webserver_port=45529
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:37839
--ranger_config_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:36121/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.217.20:36121/kms with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:36.350270 25578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:36.350530 25578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:36.350569 25578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:36.354595 25578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:47:36.354652 25578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:36.354671 25578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:36.354709 25578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:47:36.354744 25578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:47:36.360071 25578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.217.20:36121/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:37839
--ranger_config_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:37839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=45529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:36.361147 25578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:36.362159 25578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:36.370715 25593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:36.370688 25588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:36.370684 25589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:36.370791 25578 server_base.cc:1034] running on GCE node
I20260808 04:47:36.371547 25578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:36.372565 25578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:36.373728 25578 hybrid_clock.cc:648] HybridClock initialized: now 1786164456373688 us; error 51 us; skew 500 ppm
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:36.376534 25578 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:47:36.377421 25578 webserver.cc:460] Webserver started at http://127.4.217.62:45529/ using document root <none> and password file <none>
I20260808 04:47:36.377924 25578 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:36.377991 25578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:36.462839 25578 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:47:36.465160 25608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:36.466463 25578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:47:36.466626 25578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "ca82db75b3a94ee4aad8cbe646e6f600"
format_stamp: "Formatted at 2026-08-08 04:47:34 on dist-test-slave-2kcd"
server_key: "cbcff30f46ac4a39dff14041661ded5d"
server_key_iv: "5c9fe31572d4377c3a1a74e9d25dc05b"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:36.467203 25578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:36.490268 25578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:36.493479 25578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:36.493691 25578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:36.501185 25578 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:37839
I20260808 04:47:36.501233 25695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:37839 every 8 connection(s)
I20260808 04:47:36.502305 25578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:47:36.503157 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 25578
I20260808 04:47:36.503661 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:36121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:37183
--local_ip_for_outbound_sockets=127.4.217.1
--tserver_master_addrs=127.4.217.62:37839
--webserver_port=43827
--webserver_interface=127.4.217.1
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.217.20:36121/kms with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:47:36.507647 25696 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:47:36.509404 25696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Bootstrap starting.
I20260808 04:47:36.526000 25696 log.cc:824] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:36.532408 25696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:47:36.532900 25696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Bootstrap complete.
I20260808 04:47:36.536979 25696 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } }
I20260808 04:47:36.537367 25696 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca82db75b3a94ee4aad8cbe646e6f600, State: Initialized, Role: FOLLOWER
I20260808 04:47:36.537923 25696 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } }
I20260808 04:47:36.538036 25696 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:47:36.538118 25696 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:47:36.538232 25696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:47:36.540036 25696 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } }
I20260808 04:47:36.540369 25696 leader_election.cc:302] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca82db75b3a94ee4aad8cbe646e6f600; no voters:
I20260808 04:47:36.540658 25696 leader_election.cc:288] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:47:36.540813 25701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:47:36.541059 25701 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [term 2 LEADER]: Becoming Leader. State: Replica: ca82db75b3a94ee4aad8cbe646e6f600, State: Running, Role: LEADER
I20260808 04:47:36.541424 25701 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } }
I20260808 04:47:36.541862 25696 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:47:36.542917 25704 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca82db75b3a94ee4aad8cbe646e6f600. Latest consensus state: current_term: 2 leader_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } } }
I20260808 04:47:36.542946 25703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca82db75b3a94ee4aad8cbe646e6f600" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37839 } } }
I20260808 04:47:36.543052 25704 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:36.543066 25703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:36.543483 25717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:47:36.546679 25717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:47:36.547191 25696 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:47:36.547436 25717 catalog_manager.cc:1259] Loaded cluster ID: 5df821a992054349a4a34fe0f6108ce0
I20260808 04:47:36.547485 25717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:47:36.550024 25717 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:47:36.550719 25717 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Loaded TSK: 0
I20260808 04:47:36.551908 25717 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:47:36.642956 25699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:36.643271 25699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:36.643335 25699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:36.649297 25699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:36.649367 25699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:36.649451 25699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:47:36.656699 25699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.217.20:36121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:37183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=43827
--tserver_master_addrs=127.4.217.62:37839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:36.658053 25699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:36.659341 25699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:36.671361 25759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:36.671368 25763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:36.671845 25757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:36.673106 25699 server_base.cc:1034] running on GCE node
I20260808 04:47:36.673542 25699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:36.674286 25699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:36.675474 25699 hybrid_clock.cc:648] HybridClock initialized: now 1786164456675457 us; error 30 us; skew 500 ppm
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:36.681617 25699 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:47:36.682699 25699 webserver.cc:460] Webserver started at http://127.4.217.1:43827/ using document root <none> and password file <none>
I20260808 04:47:36.683399 25699 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:36.683526 25699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:36.707505 25699 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260808 04:47:36.710973 25775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:36.712568 25699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260808 04:47:36.712700 25699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "5dc00a157fb74e989b8716d54614165c"
format_stamp: "Formatted at 2026-08-08 04:47:35 on dist-test-slave-2kcd"
server_key: "243bf774df3bb058cc488fcf612492b8"
server_key_iv: "24333784fe64fb3b9b35effa1c8e7058"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:36.713428 25699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:36.742120 25699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:36.745814 25699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:36.746083 25699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:36.746850 25699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:36.747942 25699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:36.748067 25699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:36.748188 25699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:36.748224 25699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:36.759368 25699 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:37183
I20260808 04:47:36.759467 25979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:37183 every 8 connection(s)
I20260808 04:47:36.760500 25699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:47:36.764329 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 25699
I20260808 04:47:36.764881 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:36121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:39751
--local_ip_for_outbound_sockets=127.4.217.2
--tserver_master_addrs=127.4.217.62:37839
--webserver_port=42335
--webserver_interface=127.4.217.2
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.217.20:36121/kms with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:36.780690 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:36.763091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37839 (local address 127.4.217.1:53537)
0808 04:47:36.763629 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:36.763681 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:36.764601 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:36.766118 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:36.766126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:36.766473 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:36.767118 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:36.767155 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:36.768971 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:36.768975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:36.770395 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:36.770408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:36.771708 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:36.771713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:36.771905 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:36.772583 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:36.772605 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:36.774212 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:36.776862 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:36.776868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:36.776882 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:36.777230 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:36.777685 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:36.777688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:36.777691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:36.777888 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:36.778735 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:36.778742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:36.779031 (+ 289us) client_negotiation.cc:769] Sending connection context
0808 04:47:36.779232 (+ 201us) client_negotiation.cc:240] Negotiation successful
0808 04:47:36.779601 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":120,"threads_started":1}
I20260808 04:47:36.781790 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:36.762948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:53537 (local address 127.4.217.62:37839)
0808 04:47:36.765115 (+ 2167us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:36.765138 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:36.765180 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:47:36.765425 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:36.765435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:36.765718 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:36.766037 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:36.767350 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:36.768710 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:36.770614 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:36.771347 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:36.772259 (+ 912us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:36.774446 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:36.774453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:36.774470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:36.774505 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:36.776673 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:36.777379 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:36.777385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:36.777393 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:36.777538 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:36.778017 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:36.778021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:36.778023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:36.778504 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:36.778660 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:36.780587 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0808 04:47:36.780923 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1965,"thread_start_us":118,"threads_started":1}
I20260808 04:47:36.784361 25980 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37839
I20260808 04:47:36.784636 25980 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:36.785225 25980 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
W20260808 04:47:36.787477 25980 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:37839 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:47:36.787613 25980 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:36.787786 25980 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
I20260808 04:47:36.788746 25980 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:36.788964 25980 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
W20260808 04:47:36.789788 25980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:47:36.907737 25984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:36.908002 25984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:36.908078 25984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:36.912294 25984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:36.912364 25984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:36.912463 25984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:47:36.918217 25984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.217.20:36121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:39751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=42335
--tserver_master_addrs=127.4.217.62:37839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:36.919345 25984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:36.920480 25984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:36.931356 25993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:36.931381 25994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:36.931382 25997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:36.932503 25984 server_base.cc:1034] running on GCE node
I20260808 04:47:36.932966 25984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:36.933660 25984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:36.934840 25984 hybrid_clock.cc:648] HybridClock initialized: now 1786164456934810 us; error 40 us; skew 500 ppm
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:36.938228 25984 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:47:36.939065 25984 webserver.cc:460] Webserver started at http://127.4.217.2:42335/ using document root <none> and password file <none>
I20260808 04:47:36.939663 25984 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:36.939740 25984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:47:36 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:36.970947 25984 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:47:36.974363 26013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:36.975939 25984 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:47:36.976061 25984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "20c76e9a35ba401384c7fe874387eaf4"
format_stamp: "Formatted at 2026-08-08 04:47:35 on dist-test-slave-2kcd"
server_key: "a7788f034d0851b63f8d11a961ddb997"
server_key_iv: "9b61356e74979a13074a6b2f50b2b232"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:36.976838 25984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:37.001610 25984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:37.005565 25984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:37.005813 25984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:37.006659 25984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:37.007912 25984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:37.007987 25984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:37.008050 25984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:37.008086 25984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:37.020327 25984 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:39751
I20260808 04:47:37.021814 25984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:47:37.022728 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 25984
I20260808 04:47:37.023017 26210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:39751 every 8 connection(s)
I20260808 04:47:37.023250 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.217.20:36121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:39195
--local_ip_for_outbound_sockets=127.4.217.3
--tserver_master_addrs=127.4.217.62:37839
--webserver_port=41133
--webserver_interface=127.4.217.3
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.217.20:36121/kms with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:47:37 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:37.044111 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:37.027119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:33761 (local address 127.4.217.62:37839)
0808 04:47:37.027279 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:37.027285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:37.028097 (+ 812us) server_negotiation.cc:407] Connection header received
0808 04:47:37.029498 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:37.029503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:37.029569 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:37.029683 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:37.031259 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.032081 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.034028 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.034435 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.034526 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:37.038159 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:37.038166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:37.038173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:37.038215 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:37.040379 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:37.041144 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:37.041148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:37.041151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:37.041250 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:37.041689 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:37.041693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:37.041695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:37.041962 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:37.042056 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:37.043585 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0808 04:47:37.043792 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:47:37.046573 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:37.025736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37839 (local address 127.4.217.2:33761)
0808 04:47:37.027881 (+ 2145us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:37.027952 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:37.029201 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:37.029924 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:37.029934 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:37.030330 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:37.031064 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:37.031086 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.032251 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.032256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:37.033847 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:37.033861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.034759 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.034763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:37.035032 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:37.035818 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:37.035847 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:37.037921 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:37.040533 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:37.040540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:37.040560 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:37.041006 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:37.041370 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:37.041373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:37.041377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:37.041568 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:37.042134 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:37.042141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:37.043045 (+ 904us) client_negotiation.cc:769] Sending connection context
0808 04:47:37.044717 (+ 1672us) client_negotiation.cc:240] Negotiation successful
0808 04:47:37.045205 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1896,"thread_start_us":182,"threads_started":1}
I20260808 04:47:37.048537 26211 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37839
I20260808 04:47:37.048893 26211 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:37.049629 26211 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
W20260808 04:47:37.051769 26211 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:37839 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:47:37.051939 26211 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:37.052150 26211 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
I20260808 04:47:37.053524 26211 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:37.053740 26211 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
W20260808 04:47:37.054670 26211 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:47:37.177722 26214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:37.178110 26214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:37.178210 26214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:37.184098 26214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:37.184217 26214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:37.184355 26214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:47:37.191880 26214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.217.20:36121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:39195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=41133
--tserver_master_addrs=127.4.217.62:37839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:37.193214 26214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:37.194509 26214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:37.206061 26229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:37.206073 26226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:37.206558 26214 server_base.cc:1034] running on GCE node
W20260808 04:47:37.208937 26225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:37.209390 26214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:37.210129 26214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:37.211324 26214 hybrid_clock.cc:648] HybridClock initialized: now 1786164457211295 us; error 44 us; skew 500 ppm
Aug 08 04:47:37 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:37.214174 26214 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:47:37.215060 26214 webserver.cc:460] Webserver started at http://127.4.217.3:41133/ using document root <none> and password file <none>
I20260808 04:47:37.215708 26214 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:37.215793 26214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:47:37 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:37.238214 26214 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260808 04:47:37.241637 26242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:37.243172 26214 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 04:47:37.243287 26214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "20b6101ceb884a328df7bca32799218e"
format_stamp: "Formatted at 2026-08-08 04:47:36 on dist-test-slave-2kcd"
server_key: "12c81c0c214b6250610d53e13c98646f"
server_key_iv: "24e7cc384b85a0b4631280feda7d1116"
server_key_version: "kuduclusterkey@0"
I20260808 04:47:37.243968 26214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:37.282680 26214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:37.286309 26214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:37.286592 26214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:37.287343 26214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:37.288477 26214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:37.288551 26214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:37.288630 26214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:37.288668 26214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:37.303823 26214 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:39195
I20260808 04:47:37.303851 26444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:39195 every 8 connection(s)
I20260808 04:47:37.305084 26214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:47:37.308159 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 26214
Aug 08 04:47:37 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:37.321496 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:37.307065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:35159 (local address 127.4.217.62:37839)
0808 04:47:37.307196 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:37.307200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:37.308426 (+ 1226us) server_negotiation.cc:407] Connection header received
0808 04:47:37.309725 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:37.309729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:37.309806 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:37.309891 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:37.311209 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.312047 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.313623 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.313974 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.314083 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:37.317154 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:37.317160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:37.317163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:37.317215 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:37.318859 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:37.319608 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:37.319612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:37.319614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:37.319713 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:37.320205 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:37.320209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:37.320211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:37.320480 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:37.320604 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:37.321203 (+ 599us) server_negotiation.cc:299] Negotiation successful
0808 04:47:37.321323 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:47:37.322451 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:37.307534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37839 (local address 127.4.217.3:35159)
0808 04:47:37.308259 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:37.308317 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:37.309458 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:37.310239 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:37.310248 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:37.310588 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:37.311063 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:37.311077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.312224 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.312227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:37.313486 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:37.313497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.314250 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.314255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:37.314465 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:37.315066 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:37.315082 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:37.316959 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:37.318981 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:37.318987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:37.319000 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:37.319447 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:37.319837 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:37.319841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:37.319843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:37.320028 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:37.320613 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:37.320623 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:37.320947 (+ 324us) client_negotiation.cc:769] Sending connection context
0808 04:47:37.321154 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 04:47:37.321440 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":260,"threads_started":1}
I20260808 04:47:37.322669 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:37.309742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.4.217.62:37839)
0808 04:47:37.310219 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:37.310229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:37.310242 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:47:37.310273 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:37.310276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:37.310314 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:37.310411 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:37.311361 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.312070 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.313331 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:37.313695 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:37.313812 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:37.315522 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:37.315529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:37.315535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:37.315570 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:37.317316 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:37.320872 (+ 3556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:37.320878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:37.320882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:37.320968 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:37.321712 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:37.321716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:37.321717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:37.321908 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:37.322106 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:37.322328 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 04:47:37.322452 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":93,"threads_started":1}
I20260808 04:47:37.324021 26447 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37839
I20260808 04:47:37.324254 26447 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:37.324893 26447 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
W20260808 04:47:37.326678 26447 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:37839 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:47:37.326817 26447 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:37.327008 26447 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
I20260808 04:47:37.327958 26447 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:37.328145 26447 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
I20260808 04:47:37.329053 25635 ts_manager.cc:194] Registered new tserver with Master: 20b6101ceb884a328df7bca32799218e (127.4.217.3:39195)
I20260808 04:47:37.330180 25635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:35159
Aug 08 04:47:37 dist-test-slave-2kcd krb5kdc[23996](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:37.790056 25980 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:37.790416 25980 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
I20260808 04:47:37.791289 25635 ts_manager.cc:194] Registered new tserver with Master: 5dc00a157fb74e989b8716d54614165c (127.4.217.1:37183)
I20260808 04:47:37.791937 25635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:53537
Aug 08 04:47:38 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:38.054978 26211 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:38.055289 26211 heartbeater.cc:511] Master 127.4.217.62:37839 requested a full tablet report, sending...
I20260808 04:47:38.056125 25635 ts_manager.cc:194] Registered new tserver with Master: 20c76e9a35ba401384c7fe874387eaf4 (127.4.217.2:39751)
I20260808 04:47:38.057003 25635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:33761
I20260808 04:47:38.062851 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:47:38.078112 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.065383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.4.217.62:37839)
0808 04:47:38.066085 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:38.066090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:38.066487 (+ 397us) server_negotiation.cc:407] Connection header received
0808 04:47:38.066687 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:38.066691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:38.066746 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:38.066971 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:38.069306 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.070112 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.071353 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.071761 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.071858 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.073671 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:38.073675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:38.073678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:38.073704 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:38.075674 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:38.076185 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:38.076188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:38.076190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:38.076288 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:38.076653 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:38.076656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:38.076658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:38.076947 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:38.077080 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:38.077791 (+ 711us) server_negotiation.cc:299] Negotiation successful
0808 04:47:38.077924 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":92,"threads_started":1}
I20260808 04:47:38.082705 25635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:47:38.088048 25635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:47:38.124544 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.107748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:39195 (local address 127.0.0.1:43442)
0808 04:47:38.111536 (+ 3788us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:38.111593 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:38.111741 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:38.118518 (+ 6777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:38.118522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:38.118546 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:38.118725 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.118733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.120978 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.120982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.122758 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.122774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.123952 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.123957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.124080 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.124133 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 04:47:38.124233 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:47:38.124358 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3590,"thread_start_us":153,"threads_started":1}
I20260808 04:47:38.125350 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.107872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.4.217.3:39195)
0808 04:47:38.114947 (+ 7075us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:38.114954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:38.114979 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:47:38.115061 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:38.115067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:38.115268 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:38.118920 (+ 3652us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:38.119103 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.120339 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.122917 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.123773 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.124835 (+ 1062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.125023 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:38.125082 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 04:47:38.125187 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6953,"thread_start_us":73,"threads_started":1}
I20260808 04:47:38.128381 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.110780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:39751 (local address 127.0.0.1:37934)
0808 04:47:38.113217 (+ 2437us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:38.113232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:38.113347 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:38.114451 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:38.114458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:38.114487 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:38.114633 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.114644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.125641 (+ 10997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.125645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.127046 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.127066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.127945 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.127949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.128035 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.128063 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:47:38.128141 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:47:38.128218 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2371,"thread_start_us":196,"threads_started":1}
I20260808 04:47:38.128435 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.110944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.4.217.2:39751)
0808 04:47:38.113653 (+ 2709us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:38.113662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:38.113691 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:47:38.113780 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:38.113787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:38.114011 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:38.114173 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:38.117006 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.125271 (+ 8265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.127214 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.127793 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.127944 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.128116 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:38.128174 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 04:47:38.128271 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2582,"thread_start_us":130,"threads_started":1}
I20260808 04:47:38.128846 26323 tablet_service.cc:1467] Processing CreateTablet for tablet df3c3f5b26ee42b094d4fe4da1049be1 (DEFAULT_TABLE table=test-table [id=3320ee41bae24f5e857a6d85caa073dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:47:38.130221 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3c3f5b26ee42b094d4fe4da1049be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:38.131598 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.109894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:37183 (local address 127.0.0.1:45716)
0808 04:47:38.112899 (+ 3005us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:38.112914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:38.113014 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:38.122987 (+ 9973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:38.122990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:38.123013 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:38.123127 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.123135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.128920 (+ 5785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.128924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.130050 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.130064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.131192 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.131196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.131300 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.131328 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:47:38.131382 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 04:47:38.131462 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2929,"thread_start_us":209,"threads_started":1}
I20260808 04:47:38.132310 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.110063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.4.217.1:37183)
0808 04:47:38.115624 (+ 5561us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:38.115632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:38.115656 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:47:38.116575 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:38.116583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:38.116816 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:38.123305 (+ 6489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:38.123523 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.124601 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.130224 (+ 5623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.131031 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.131839 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.132002 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:38.132059 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 04:47:38.132156 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5420,"thread_start_us":64,"threads_started":1}
I20260808 04:47:38.131978 26096 tablet_service.cc:1467] Processing CreateTablet for tablet df3c3f5b26ee42b094d4fe4da1049be1 (DEFAULT_TABLE table=test-table [id=3320ee41bae24f5e857a6d85caa073dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:47:38.133068 26096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3c3f5b26ee42b094d4fe4da1049be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:38.135121 25853 tablet_service.cc:1467] Processing CreateTablet for tablet df3c3f5b26ee42b094d4fe4da1049be1 (DEFAULT_TABLE table=test-table [id=3320ee41bae24f5e857a6d85caa073dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:47:38.136224 25853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3c3f5b26ee42b094d4fe4da1049be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:38.143996 26507 tablet_bootstrap.cc:492] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c: Bootstrap starting.
I20260808 04:47:38.144981 26506 tablet_bootstrap.cc:492] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: Bootstrap starting.
I20260808 04:47:38.146587 26507 tablet_bootstrap.cc:654] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:38.147171 26506 tablet_bootstrap.cc:654] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:38.147502 26507 log.cc:824] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:38.147989 26506 log.cc:824] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:38.148765 26508 tablet_bootstrap.cc:492] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4: Bootstrap starting.
I20260808 04:47:38.150621 26508 tablet_bootstrap.cc:654] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:38.151276 26508 log.cc:824] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:38.153467 26506 tablet_bootstrap.cc:492] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: No bootstrap required, opened a new log
I20260808 04:47:38.153695 26506 ts_tablet_manager.cc:1397] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260808 04:47:38.156914 26506 raft_consensus.cc:348] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } }
I20260808 04:47:38.157208 26506 raft_consensus.cc:374] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:38.157264 26506 raft_consensus.cc:729] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b6101ceb884a328df7bca32799218e, State: Initialized, Role: FOLLOWER
I20260808 04:47:38.157773 26506 consensus_queue.cc:260] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } }
I20260808 04:47:38.158684 26508 tablet_bootstrap.cc:492] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4: No bootstrap required, opened a new log
I20260808 04:47:38.158865 26506 ts_tablet_manager.cc:1428] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260808 04:47:38.159132 26508 ts_tablet_manager.cc:1397] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260808 04:47:38.159242 26447 heartbeater.cc:503] Master 127.4.217.62:37839 was elected leader, sending a full tablet report...
I20260808 04:47:38.164896 26507 tablet_bootstrap.cc:492] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c: No bootstrap required, opened a new log
I20260808 04:47:38.165161 26507 ts_tablet_manager.cc:1397] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260808 04:47:38.167385 26508 raft_consensus.cc:348] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } }
I20260808 04:47:38.167667 26508 raft_consensus.cc:374] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:38.167717 26508 raft_consensus.cc:729] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c76e9a35ba401384c7fe874387eaf4, State: Initialized, Role: FOLLOWER
I20260808 04:47:38.168226 26508 consensus_queue.cc:260] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } }
I20260808 04:47:38.169538 26508 ts_tablet_manager.cc:1428] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260808 04:47:38.169183 26507 raft_consensus.cc:348] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } }
I20260808 04:47:38.169661 26507 raft_consensus.cc:374] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:38.169725 26507 raft_consensus.cc:729] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc00a157fb74e989b8716d54614165c, State: Initialized, Role: FOLLOWER
I20260808 04:47:38.169907 26211 heartbeater.cc:503] Master 127.4.217.62:37839 was elected leader, sending a full tablet report...
I20260808 04:47:38.170171 26507 consensus_queue.cc:260] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } }
I20260808 04:47:38.171641 25980 heartbeater.cc:503] Master 127.4.217.62:37839 was elected leader, sending a full tablet report...
I20260808 04:47:38.176666 26507 ts_tablet_manager.cc:1428] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260808 04:47:38.243677 26516 raft_consensus.cc:482] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:47:38.243922 26516 raft_consensus.cc:504] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } }
I20260808 04:47:38.245457 26516 leader_election.cc:288] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b6101ceb884a328df7bca32799218e (127.4.217.3:39195), 5dc00a157fb74e989b8716d54614165c (127.4.217.1:37183)
I20260808 04:47:38.251340 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.245843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:55219 (local address 127.4.217.3:39195)
0808 04:47:38.245980 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:38.245985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:38.246238 (+ 253us) server_negotiation.cc:407] Connection header received
0808 04:47:38.246457 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:38.246461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:38.246512 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:38.246557 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:38.246995 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.247751 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.249759 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.250451 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.250547 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.250631 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:38.251069 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 04:47:38.251165 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:47:38.251987 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.245712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:39195 (local address 127.4.217.2:55219)
0808 04:47:38.246152 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:38.246166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:38.246316 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:38.246697 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:38.246700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:38.246717 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:38.246863 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.246870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.247891 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.247894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.249617 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.249628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.250773 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.250779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.250904 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.250933 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:47:38.251743 (+ 810us) client_negotiation.cc:240] Negotiation successful
0808 04:47:38.251825 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":121,"threads_started":1}
I20260808 04:47:38.253170 26360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3c3f5b26ee42b094d4fe4da1049be1" candidate_uuid: "20c76e9a35ba401384c7fe874387eaf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b6101ceb884a328df7bca32799218e" is_pre_election: true
I20260808 04:47:38.253749 26360 raft_consensus.cc:2454] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c76e9a35ba401384c7fe874387eaf4 in term 0.
I20260808 04:47:38.254822 26015 leader_election.cc:302] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b6101ceb884a328df7bca32799218e, 20c76e9a35ba401384c7fe874387eaf4; no voters:
I20260808 04:47:38.255339 26516 raft_consensus.cc:2781] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:47:38.255527 26516 raft_consensus.cc:482] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:47:38.255606 26516 raft_consensus.cc:3037] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:38.257366 26516 raft_consensus.cc:504] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } }
I20260808 04:47:38.257701 26516 leader_election.cc:288] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [CANDIDATE]: Term 1 election: Requested vote from peers 20b6101ceb884a328df7bca32799218e (127.4.217.3:39195), 5dc00a157fb74e989b8716d54614165c (127.4.217.1:37183)
I20260808 04:47:38.258163 26360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3c3f5b26ee42b094d4fe4da1049be1" candidate_uuid: "20c76e9a35ba401384c7fe874387eaf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b6101ceb884a328df7bca32799218e"
I20260808 04:47:38.258273 26360 raft_consensus.cc:3037] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:38.259464 26360 raft_consensus.cc:2454] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c76e9a35ba401384c7fe874387eaf4 in term 1.
I20260808 04:47:38.259873 26015 leader_election.cc:302] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b6101ceb884a328df7bca32799218e, 20c76e9a35ba401384c7fe874387eaf4; no voters:
I20260808 04:47:38.260072 26516 raft_consensus.cc:2781] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:47:38.260447 26516 raft_consensus.cc:686] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 1 LEADER]: Becoming Leader. State: Replica: 20c76e9a35ba401384c7fe874387eaf4, State: Running, Role: LEADER
I20260808 04:47:38.260473 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.246537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:52313 (local address 127.4.217.1:37183)
0808 04:47:38.246657 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:38.246662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:38.246896 (+ 234us) server_negotiation.cc:407] Connection header received
0808 04:47:38.247121 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:38.247128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:38.247185 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:38.247240 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:38.247681 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.248406 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.259074 (+ 10668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.259692 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.259777 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.259840 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:38.260247 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 04:47:38.260334 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:47:38.260821 26516 consensus_queue.cc:237] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } }
I20260808 04:47:38.261094 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.246123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:37183 (local address 127.4.217.2:52313)
0808 04:47:38.246802 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:38.246816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:38.246976 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:38.247393 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:38.247396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:38.247413 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:38.247549 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.247556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.248770 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.248773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.258902 (+ 10129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:38.258914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.259966 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.259970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:38.260075 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.260101 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:47:38.260662 (+ 561us) client_negotiation.cc:240] Negotiation successful
0808 04:47:38.260736 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":104,"threads_started":1}
I20260808 04:47:38.261660 25896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3c3f5b26ee42b094d4fe4da1049be1" candidate_uuid: "20c76e9a35ba401384c7fe874387eaf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc00a157fb74e989b8716d54614165c" is_pre_election: true
I20260808 04:47:38.261984 25896 raft_consensus.cc:2454] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c76e9a35ba401384c7fe874387eaf4 in term 0.
I20260808 04:47:38.262233 25894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3c3f5b26ee42b094d4fe4da1049be1" candidate_uuid: "20c76e9a35ba401384c7fe874387eaf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc00a157fb74e989b8716d54614165c"
I20260808 04:47:38.262370 25894 raft_consensus.cc:3037] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c [term 0 FOLLOWER]: Advancing to term 1
W20260808 04:47:38.262604 25981 tablet.cc:2369] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:47:38.263600 25894 raft_consensus.cc:2454] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c76e9a35ba401384c7fe874387eaf4 in term 1.
I20260808 04:47:38.265609 25636 catalog_manager.cc:5495] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20c76e9a35ba401384c7fe874387eaf4 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "20c76e9a35ba401384c7fe874387eaf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20c76e9a35ba401384c7fe874387eaf4" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 39751 } health_report { overall_health: HEALTHY } } }
W20260808 04:47:38.278563 26212 tablet.cc:2369] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:47:38.306703 26448 tablet.cc:2369] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 04:47:38 dist-test-slave-2kcd krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for HTTP/127.4.217.20@KRBTEST.COM
I20260808 04:47:38.315893 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.302386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.4.217.2:39751)
0808 04:47:38.302546 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:38.302551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:38.302976 (+ 425us) server_negotiation.cc:407] Connection header received
0808 04:47:38.303078 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:38.303083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:38.311107 (+ 8024us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:38.311223 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:47:38.311859 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.312600 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.313838 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:38.314153 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:38.314245 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:38.314634 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:47:38.315143 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:47:38.315250 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:38.315659 (+ 409us) server_negotiation.cc:299] Negotiation successful
0808 04:47:38.315721 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:47:38.327215 26360 raft_consensus.cc:1261] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [term 1 FOLLOWER]: Refusing update from remote peer 20c76e9a35ba401384c7fe874387eaf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:47:38.327289 25894 raft_consensus.cc:1261] T df3c3f5b26ee42b094d4fe4da1049be1 P 5dc00a157fb74e989b8716d54614165c [term 1 FOLLOWER]: Refusing update from remote peer 20c76e9a35ba401384c7fe874387eaf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:47:38.327919 26516 consensus_queue.cc:1035] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc00a157fb74e989b8716d54614165c" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 37183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:47:38.328177 26520 consensus_queue.cc:1035] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b6101ceb884a328df7bca32799218e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:47:38.335445 26526 mvcc.cc:205] Tried to move back new op lower bound from 7316129621299163136 to 7316129621040545792. Current Snapshot: MvccSnapshot[applied={T|T < 7316129621299163136}]
I20260808 04:47:38.346503 25635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:47:38.346617 26527 mvcc.cc:205] Tried to move back new op lower bound from 7316129621299163136 to 7316129621040545792. Current Snapshot: MvccSnapshot[applied={T|T < 7316129621299163136}]
I20260808 04:47:38.346709 25635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:47:38.350044 25635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca82db75b3a94ee4aad8cbe646e6f600: Sending DeleteTablet for 3 replicas of tablet df3c3f5b26ee42b094d4fe4da1049be1
I20260808 04:47:38.350829 25853 tablet_service.cc:1514] Processing DeleteTablet for tablet df3c3f5b26ee42b094d4fe4da1049be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:38 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:45716
I20260808 04:47:38.351063 26096 tablet_service.cc:1514] Processing DeleteTablet for tablet df3c3f5b26ee42b094d4fe4da1049be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:38 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:37934
I20260808 04:47:38.351647 26323 tablet_service.cc:1514] Processing DeleteTablet for tablet df3c3f5b26ee42b094d4fe4da1049be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:38 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:43442
I20260808 04:47:38.352012 26322 tablet_service.cc:1514] Processing DeleteTablet for tablet df3c3f5b26ee42b094d4fe4da1049be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:38 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:43442
I20260808 04:47:38.352053 26547 tablet_replica.cc:330] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: stopping tablet replica
I20260808 04:47:38.352334 26547 raft_consensus.cc:2229] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:47:38.352675 26547 raft_consensus.cc:2258] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:47:38.352973 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 25699
I20260808 04:47:38.387329 26545 tablet_replica.cc:330] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4: stopping tablet replica
I20260808 04:47:38.389566 26547 ts_tablet_manager.cc:1905] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:47:38.392290 26545 raft_consensus.cc:2229] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:47:38.394241 26545 raft_consensus.cc:2258] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:47:38.395762 26547 ts_tablet_manager.cc:1918] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:47:38.395953 26547 log.cc:1196] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/df3c3f5b26ee42b094d4fe4da1049be1
I20260808 04:47:38.396376 26547 ts_tablet_manager.cc:1939] T df3c3f5b26ee42b094d4fe4da1049be1 P 20b6101ceb884a328df7bca32799218e: Deleting consensus metadata
I20260808 04:47:38.397958 25611 catalog_manager.cc:4841] TS 20b6101ceb884a328df7bca32799218e (127.4.217.3:39195): tablet df3c3f5b26ee42b094d4fe4da1049be1 (table test-table [id=3320ee41bae24f5e857a6d85caa073dc]) successfully deleted
W20260808 04:47:38.399266 25611 catalog_manager.cc:4808] TS 20b6101ceb884a328df7bca32799218e (127.4.217.3:39195): delete failed for tablet df3c3f5b26ee42b094d4fe4da1049be1 because the tablet was not found. No further retry: Not found: Tablet not found: df3c3f5b26ee42b094d4fe4da1049be1
I20260808 04:47:38.413341 26545 ts_tablet_manager.cc:1905] T df3c3f5b26ee42b094d4fe4da1049be1 P 20c76e9a35ba401384c7fe874387eaf4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 04:47:38.417312 25611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:37183 (error 108)
I20260808 04:47:38.419356 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.419008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:37183 (local address 127.0.0.1:45728)
0808 04:47:38.419171 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:38.419258 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:47:38.419582 25611 catalog_manager.cc:4584] TS 5dc00a157fb74e989b8716d54614165c (127.4.217.1:37183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df3c3f5b26ee42b094d4fe4da1049be1: Network error: Client connection negotiation failed: client connection to 127.4.217.1:37183: connect: Connection refused (error 111)
I20260808 04:47:38.423213 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 25984
W20260808 04:47:38.460309 26499 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:47:38.459566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:39751 (local address 127.0.0.1:37946)
0808 04:47:38.459789 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:38.459806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:38.459907 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:38.460136 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.2:39751: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 04:47:38.461040 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 26214
I20260808 04:47:38.472786 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.472425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:37183 (local address 127.0.0.1:45742)
0808 04:47:38.472599 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:38.472673 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 04:47:38.473681 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:38.473416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:37183 (local address 127.0.0.1:45754)
0808 04:47:38.473562 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:38.473608 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 04:47:38.477988 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 25578
I20260808 04:47:38.497574 4964 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 04:47:38.844120 4964 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 04:47:38.844259 4964 mini_ranger.cc:66] Stopping Ranger...
I20260808 04:47:39.199697 4964 mini_ranger.cc:68] Stopped Ranger
2026-08-08 04:47:39.199 UTC [24043] LOG: received smart shutdown request
2026-08-08 04:47:39.204 UTC [24043] LOG: background worker "logical replication launcher" (PID 24058) exited with exit code 1
2026-08-08 04:47:39.204 UTC [24048] LOG: shutting down
2026-08-08 04:47:39.359 UTC [24043] LOG: database system is shut down
2026-08-08T04:47:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48656 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:47:39 dist-test-slave-2kcd krb5kdc[26566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:47:39 dist-test-slave-2kcd krb5kdc[26566](info): set up 2 sockets
Aug 08 04:47:39 dist-test-slave-2kcd krb5kdc[26566](info): commencing operation
krb5kdc: starting...
W20260808 04:47:40.736197 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:47:40 dist-test-slave-2kcd krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:47:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:47:40.863431 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:36009
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:36009,127.4.217.61:38513,127.4.217.60:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:40.974610 26587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:40.974869 26587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:40.974908 26587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:40.978917 26587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:47:40.978971 26587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:40.978989 26587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:40.979005 26587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:47:40.979049 26587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:47:40.984692 26587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:36009,127.4.217.61:38513,127.4.217.60:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:40.985770 26587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:40.986797 26587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:40.995419 26597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:40.995445 26598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:40.995474 26600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:40.995682 26587 server_base.cc:1034] running on GCE node
I20260808 04:47:40.996245 26587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:40.997527 26587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:40.998745 26587 hybrid_clock.cc:648] HybridClock initialized: now 1786164460998713 us; error 41 us; skew 500 ppm
Aug 08 04:47:40 dist-test-slave-2kcd krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:41.001648 26587 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:47:41.002542 26587 webserver.cc:460] Webserver started at http://127.4.217.62:42729/ using document root <none> and password file <none>
I20260808 04:47:41.003108 26587 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:41.003198 26587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:41.003434 26587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:47:41.005599 26587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "731e707b6edc41dd96f90ebb7a5ec3fd"
format_stamp: "Formatted at 2026-08-08 04:47:40 on dist-test-slave-2kcd"
server_key: "ee34ce225f3a5e7736a6e0df267b344f"
server_key_iv: "89e33673ed050ffbf1d4f641d0ce53cb"
server_key_version: "encryptionkey@0"
I20260808 04:47:41.006177 26587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "731e707b6edc41dd96f90ebb7a5ec3fd"
format_stamp: "Formatted at 2026-08-08 04:47:40 on dist-test-slave-2kcd"
server_key: "ee34ce225f3a5e7736a6e0df267b344f"
server_key_iv: "89e33673ed050ffbf1d4f641d0ce53cb"
server_key_version: "encryptionkey@0"
I20260808 04:47:41.009873 26587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:47:41.012606 26616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:41.013895 26587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:47:41.014050 26587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "731e707b6edc41dd96f90ebb7a5ec3fd"
format_stamp: "Formatted at 2026-08-08 04:47:40 on dist-test-slave-2kcd"
server_key: "ee34ce225f3a5e7736a6e0df267b344f"
server_key_iv: "89e33673ed050ffbf1d4f641d0ce53cb"
server_key_version: "encryptionkey@0"
I20260808 04:47:41.014175 26587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:41.029692 26587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:41.033324 26587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:41.033533 26587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:41.040868 26587 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:36009
I20260808 04:47:41.040891 26697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:36009 every 8 connection(s)
I20260808 04:47:41.041827 26587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:47:41.044523 26698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:41.048264 26698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:47:41.049623 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 26587
I20260808 04:47:41.049782 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:47:41.050109 4964 external_mini_cluster.cc:1428] Setting key c41ee4087510745d1c8ccaf50c511e65
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.61@KRBTEST.COM" created.
I20260808 04:47:41.064169 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.050607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:36009 (local address 127.0.0.1:57060)
0808 04:47:41.051161 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.051197 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:41.051987 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:41.052735 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:41.052743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:41.053269 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:41.053659 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.053682 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.054795 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.054802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.056072 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.056083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.057064 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.057067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.057249 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.057839 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:41.057857 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:41.059353 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:41.061239 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.061248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.061255 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.061480 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.061836 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.061839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.061840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.061951 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.062734 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:41.062740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:41.062908 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 04:47:41.063124 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 04:47:41.063496 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":124,"thread_start_us":150,"threads_started":1}
I20260808 04:47:41.064183 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.050606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.4.217.62:36009)
0808 04:47:41.051006 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.051024 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.051302 (+ 278us) server_negotiation.cc:407] Connection header received
0808 04:47:41.052356 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.052361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.052530 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.052676 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.053823 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.054651 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.056200 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.056897 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.057102 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.059507 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.059517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.059533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.059565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.061140 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.061590 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.061596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.061602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.061745 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.062035 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.062037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.062038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.062577 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.062718 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.063182 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.063389 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
I20260808 04:47:41.066613 26698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:47:41.067324 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.067099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38513 (local address 127.0.0.1:46114)
0808 04:47:41.067209 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.067274 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 04:47:41.067622 26618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
I20260808 04:47:41.068497 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.068302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38513 (local address 127.0.0.1:46130)
0808 04:47:41.068409 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.068451 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:47:41.068924 26698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.61:38513: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab.
Entry for principal kudu/127.4.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab.
I20260808 04:47:41.079034 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:38513
--webserver_interface=127.4.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab
--principal=kudu/127.4.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:36009,127.4.217.61:38513,127.4.217.60:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:47:41.118148 26698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } attempt: 1
I20260808 04:47:41.119052 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.118728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38513 (local address 127.0.0.1:46142)
0808 04:47:41.118906 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.118965 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 04:47:41.119894 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.119698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38513 (local address 127.0.0.1:46154)
0808 04:47:41.119801 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.119844 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:47:41.120100 26698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.61:38513: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
I20260808 04:47:41.188253 26698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } attempt: 2
I20260808 04:47:41.189234 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.188877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38513 (local address 127.0.0.1:46156)
0808 04:47:41.189063 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.189136 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 04:47:41.190016 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.189792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38513 (local address 127.0.0.1:46158)
0808 04:47:41.189906 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.189953 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:47:41.190137 26698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.61:38513: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38513: connect: Connection refused (error 111)
W20260808 04:47:41.194028 26714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:41.194295 26714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:41.194413 26714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:41.198421 26714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:47:41.198503 26714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:41.198555 26714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:41.198601 26714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:47:41.198661 26714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:47:41.203843 26714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:36009,127.4.217.61:38513,127.4.217.60:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab
--principal=kudu/127.4.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:41.204921 26714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:41.205931 26714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:41.214531 26725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:41.214537 26723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:41.214545 26721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:41.214939 26714 server_base.cc:1034] running on GCE node
I20260808 04:47:41.215481 26714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:41.216504 26714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:41.217707 26714 hybrid_clock.cc:648] HybridClock initialized: now 1786164461217674 us; error 42 us; skew 500 ppm
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164461, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:41.220468 26714 init.cc:377] Logged in from keytab as kudu/127.4.217.61@KRBTEST.COM (short username kudu)
I20260808 04:47:41.221371 26714 webserver.cc:460] Webserver started at http://127.4.217.61:44519/ using document root <none> and password file <none>
I20260808 04:47:41.221908 26714 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:41.221971 26714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:41.222246 26714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:47:41.224217 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/data/instance:
uuid: "7e4d4db878574ee59621307d0859bc3c"
format_stamp: "Formatted at 2026-08-08 04:47:41 on dist-test-slave-2kcd"
server_key: "38696f85e91a3ac84db87e25a3ad15e5"
server_key_iv: "9d43e9d49c4f5917dcb2534ca1a074c9"
server_key_version: "encryptionkey@0"
I20260808 04:47:41.224855 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/wal/instance:
uuid: "7e4d4db878574ee59621307d0859bc3c"
format_stamp: "Formatted at 2026-08-08 04:47:41 on dist-test-slave-2kcd"
server_key: "38696f85e91a3ac84db87e25a3ad15e5"
server_key_iv: "9d43e9d49c4f5917dcb2534ca1a074c9"
server_key_version: "encryptionkey@0"
I20260808 04:47:41.228313 26714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:47:41.230628 26736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:41.231760 26714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:47:41.231910 26714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/wal
uuid: "7e4d4db878574ee59621307d0859bc3c"
format_stamp: "Formatted at 2026-08-08 04:47:41 on dist-test-slave-2kcd"
server_key: "38696f85e91a3ac84db87e25a3ad15e5"
server_key_iv: "9d43e9d49c4f5917dcb2534ca1a074c9"
server_key_version: "encryptionkey@0"
I20260808 04:47:41.232016 26714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:41.246174 26714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:41.248873 26714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:41.249065 26714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:41.258625 26714 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:38513
I20260808 04:47:41.258688 26831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.61:38513 every 8 connection(s)
I20260808 04:47:41.259950 26714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
I20260808 04:47:41.263860 26834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:41.268623 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 26714
I20260808 04:47:41.268739 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-1/wal/instance
I20260808 04:47:41.269075 4964 external_mini_cluster.cc:1428] Setting key 124345afc33010e26792540f89873fcf
I20260808 04:47:41.269109 26834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.217.60@KRBTEST.COM; defaulting to no policy
I20260808 04:47:41.281283 26698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.60@KRBTEST.COM" created.
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164461, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
I20260808 04:47:41.290230 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.271900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:36009 (local address 127.0.0.1:57074)
0808 04:47:41.272441 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.272485 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:41.273715 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:41.274592 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:41.274607 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:41.275150 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:41.275850 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.275865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.276701 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.276704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.278183 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.278195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.278992 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.278996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.279193 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.279890 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:41.279910 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:41.284051 (+ 4141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:41.286851 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.286860 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.286880 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.287274 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.287606 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.287609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.287611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.287768 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.288304 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:41.288313 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:41.288641 (+ 328us) client_negotiation.cc:769] Sending connection context
0808 04:47:41.288895 (+ 254us) client_negotiation.cc:240] Negotiation successful
0808 04:47:41.289252 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":160,"threads_started":1}
I20260808 04:47:41.290989 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.272998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.4.217.62:36009)
0808 04:47:41.273119 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.273124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.273143 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:47:41.273978 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.273985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.274069 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.274132 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.276012 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.276555 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.278354 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.278830 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.278942 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.284207 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.284211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.284213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.284239 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.286722 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.287396 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.287400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.287402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.287499 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.287889 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.287895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.287900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.288170 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.288301 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.290726 (+ 2425us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.290868 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 04:47:41.292128 26834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:47:41.293037 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.281796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38513 (local address 127.0.0.1:46170)
0808 04:47:41.281941 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.281972 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:41.282057 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:41.282668 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:41.282672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:41.282888 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:41.283021 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.283029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.284422 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.284426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.285713 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.285730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.286594 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.286600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.286736 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.287217 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:41.287238 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:41.288705 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:41.291146 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.291151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.291154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.291415 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.291803 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.291806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.291809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.291908 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.292476 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:41.292482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:41.292600 (+ 118us) client_negotiation.cc:769] Sending connection context
0808 04:47:41.292714 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 04:47:41.292869 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 04:47:41.293871 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.281890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.4.217.61:38513)
0808 04:47:41.282242 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.282257 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.282284 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:47:41.282327 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.282333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.282508 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.282681 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.283195 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.284289 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.285874 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.286445 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.286581 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.288906 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.288915 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.288929 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.288952 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.291007 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.291578 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.291585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.291587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.291676 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.292039 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.292042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.292047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.292347 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.292494 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.293510 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.293682 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260808 04:47:41.295687 26698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:47:41.296448 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.296154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38212)
0808 04:47:41.296269 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.296323 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 04:47:41.297465 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.297146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38224)
0808 04:47:41.297303 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.297353 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260808 04:47:41.297616 26698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:35707: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164461, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
I20260808 04:47:41.301915 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.292714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38513 (local address 127.0.0.1:46180)
0808 04:47:41.292956 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.292970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:41.293038 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:41.293429 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:41.293432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:41.293661 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:41.293769 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.293776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.294704 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.294707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.295222 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.295230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.296172 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.296176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.296293 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.296746 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:41.296758 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:41.298241 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:41.300368 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.300371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.300373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.300559 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.300873 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.300876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.300878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.300961 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.301408 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:41.301411 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:41.301499 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 04:47:41.301609 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:47:41.301746 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":2}
I20260808 04:47:41.301949 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.292836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.4.217.61:38513)
0808 04:47:41.293181 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.293193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.293209 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:47:41.293243 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.293245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.293293 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.293385 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.293907 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.294591 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.295339 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.296058 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.296177 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.298810 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.298818 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.298823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.298850 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.300258 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.300658 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.300663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.300665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.300745 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.301059 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.301064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.301066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.301305 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.301408 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.301655 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.301799 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab.
Entry for principal kudu/127.4.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab.
I20260808 04:47:41.302963 26834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:47:41.303716 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.303404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38230)
0808 04:47:41.303552 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.303642 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 04:47:41.303844 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:35707
--webserver_interface=127.4.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab
--principal=kudu/127.4.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:36009,127.4.217.61:38513,127.4.217.60:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:41.303972 26740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
I20260808 04:47:41.304993 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.304662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38246)
0808 04:47:41.304852 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.304913 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:47:41.305147 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:35707: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
I20260808 04:47:41.328840 26698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } attempt: 1
I20260808 04:47:41.329666 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.329356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38248)
0808 04:47:41.329508 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.329583 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 04:47:41.330438 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.330216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38260)
0808 04:47:41.330329 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.330380 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:47:41.330585 26698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:35707: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
I20260808 04:47:41.354413 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } attempt: 1
I20260808 04:47:41.355341 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.355041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38270)
0808 04:47:41.355198 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.355259 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 04:47:41.356130 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.355914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38276)
0808 04:47:41.356024 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.356077 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:47:41.356261 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:35707: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
I20260808 04:47:41.405743 26698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } attempt: 2
I20260808 04:47:41.406648 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.406339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38282)
0808 04:47:41.406499 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.406565 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 04:47:41.407413 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.407187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38288)
0808 04:47:41.407301 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.407351 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:47:41.407550 26698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:35707: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
W20260808 04:47:41.419991 26849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:41.420275 26849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:41.420378 26849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:41.424402 26849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:47:41.424482 26849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260808 04:47:41.424415 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } attempt: 2
W20260808 04:47:41.424528 26849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:41.424587 26849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:47:41.424626 26849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:47:41.425350 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.425032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38298)
0808 04:47:41.425207 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.425275 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 04:47:41.426096 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.425873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38302)
0808 04:47:41.425985 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.426039 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:47:41.426227 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:35707: Network error: Client connection negotiation failed: client connection to 127.4.217.60:35707: connect: Connection refused (error 111)
I20260808 04:47:41.430186 26849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:36009,127.4.217.61:38513,127.4.217.60:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab
--principal=kudu/127.4.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:41.431239 26849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:41.432263 26849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:47:41.441016 26849 server_base.cc:1034] running on GCE node
W20260808 04:47:41.441006 26871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:41.440984 26864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:41.441087 26861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:41.441847 26849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:41.442830 26849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:41.443993 26849 hybrid_clock.cc:648] HybridClock initialized: now 1786164461443980 us; error 26 us; skew 500 ppm
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164461, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:41.446771 26849 init.cc:377] Logged in from keytab as kudu/127.4.217.60@KRBTEST.COM (short username kudu)
I20260808 04:47:41.447636 26849 webserver.cc:460] Webserver started at http://127.4.217.60:32911/ using document root <none> and password file <none>
I20260808 04:47:41.448175 26849 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:41.448236 26849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:41.448472 26849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:47:41.450603 26849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/data/instance:
uuid: "8e53323f0f2b45e39af35eda94880933"
format_stamp: "Formatted at 2026-08-08 04:47:41 on dist-test-slave-2kcd"
server_key: "843eebed724189e9a36a16f046103bb5"
server_key_iv: "cfe02ebff2b71086a96e5ea754f44406"
server_key_version: "encryptionkey@0"
I20260808 04:47:41.451175 26849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/wal/instance:
uuid: "8e53323f0f2b45e39af35eda94880933"
format_stamp: "Formatted at 2026-08-08 04:47:41 on dist-test-slave-2kcd"
server_key: "843eebed724189e9a36a16f046103bb5"
server_key_iv: "cfe02ebff2b71086a96e5ea754f44406"
server_key_version: "encryptionkey@0"
I20260808 04:47:41.454813 26849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:47:41.457258 26883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:41.458407 26849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:47:41.458544 26849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/wal
uuid: "8e53323f0f2b45e39af35eda94880933"
format_stamp: "Formatted at 2026-08-08 04:47:41 on dist-test-slave-2kcd"
server_key: "843eebed724189e9a36a16f046103bb5"
server_key_iv: "cfe02ebff2b71086a96e5ea754f44406"
server_key_version: "encryptionkey@0"
I20260808 04:47:41.458648 26849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:41.469158 26849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:41.471805 26849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:41.472014 26849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:41.479256 26849 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.60:35707
I20260808 04:47:41.479285 26969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.60:35707 every 8 connection(s)
I20260808 04:47:41.480156 26849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
I20260808 04:47:41.480489 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 26849
I20260808 04:47:41.480628 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164296366965-4964-0/minicluster-data/master-2/wal/instance
I20260808 04:47:41.480969 4964 external_mini_cluster.cc:1428] Setting key ae14c1c7586ba3c389403cda6c3a119f
I20260808 04:47:41.483484 26970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:41.488004 26970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164461, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:41.495909 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.482316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.4.217.60:35707)
0808 04:47:41.482725 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.482743 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.482785 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:47:41.483620 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.483643 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.484022 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.484373 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.485119 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.486573 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.487445 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.488085 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.488242 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.491016 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.491021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.491030 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.491062 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.492892 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.493354 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.493360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.493366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.493497 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.493801 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.493807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.493812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.494178 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.494416 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.494704 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.495017 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20260808 04:47:41.499455 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.489961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.4.217.62:36009)
0808 04:47:41.490138 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.490145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.490377 (+ 232us) server_negotiation.cc:407] Connection header received
0808 04:47:41.490610 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.490616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.490682 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.490779 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.491480 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.492271 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.493317 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.493645 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.493721 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.496015 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.496018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.496021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.496056 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.497588 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.498069 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.498072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.498073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.498171 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.498516 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.498519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.498521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.498721 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.498839 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.499186 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.499297 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:47:41.499475 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.489873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:36009 (local address 127.0.0.1:57080)
0808 04:47:41.490259 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.490305 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:41.490462 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:41.490830 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:41.490834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:41.491152 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:41.491321 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.491331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.492406 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.492409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.493163 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.493177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.493758 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.493763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.493866 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.494382 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:41.494394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:41.495868 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:41.497703 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.497713 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.497718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.497974 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.498296 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.498299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.498301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.498408 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.498840 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:41.498847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:41.498954 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 04:47:41.499128 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 04:47:41.499298 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":162,"threads_started":1}
I20260808 04:47:41.500479 26970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164461, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
I20260808 04:47:41.506714 26698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } attempt: 3
I20260808 04:47:41.507889 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.498251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.4.217.62:36009)
0808 04:47:41.498608 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.498615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.498626 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:47:41.498655 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.498657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.498689 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.498790 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.499395 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.500202 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.501203 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.501605 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.501711 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.504156 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.504161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.504163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.504188 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.505719 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.506140 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.506143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.506145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.506259 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.506545 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.506548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.506550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.506713 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.506930 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.507578 (+ 648us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.507711 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":111,"threads_started":1}
I20260808 04:47:41.509603 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.501126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.4.217.61:38513)
0808 04:47:41.501272 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.501275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.501308 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:47:41.501355 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.501358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.501408 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.501485 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.501947 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.502792 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.503551 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.503942 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.504073 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.505988 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.505992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.505994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.506019 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.507691 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.508148 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.508152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.508154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.508251 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.508604 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.508609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.508612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.508902 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.509005 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.509313 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.509434 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:47:41.510062 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.500991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38513 (local address 127.0.0.1:46192)
0808 04:47:41.501179 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.501193 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:41.501261 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:41.501521 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:41.501524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:41.501698 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:41.501804 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.501810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.502915 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.502917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.503420 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.503428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.504088 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.504092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.504186 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.504555 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:41.504566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:41.505853 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:41.507766 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.507770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.507775 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.508021 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.508370 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.508373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.508375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.508469 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.509000 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:41.509006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:41.509105 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 04:47:41.509741 (+ 636us) client_negotiation.cc:240] Negotiation successful
0808 04:47:41.509877 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260808 04:47:41.510818 26970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
I20260808 04:47:41.516593 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.507139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38312)
0808 04:47:41.507273 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.507287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:41.507354 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:41.507702 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:41.507705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:41.507855 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:41.507948 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.507974 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.509634 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.509638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.510343 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.510354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.511061 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.511065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.511152 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.511593 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:41.511607 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:41.513034 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:41.515012 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.515016 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.515018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.515285 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.515593 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.515595 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.515596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.515695 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.516103 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:41.516106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:41.516214 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 04:47:41.516319 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:47:41.516441 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:47:41.517177 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.507381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.4.217.60:35707)
0808 04:47:41.507485 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.507489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.507501 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:47:41.507535 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.507538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.507579 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.507667 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.508318 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.509521 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.510479 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.510855 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.510951 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.513178 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.513182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.513184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.513207 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.514896 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.515381 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.515383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.515384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.515473 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.515803 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.515808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.515811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.516000 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.516074 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.516909 (+ 835us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.517053 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164461, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
I20260808 04:47:41.517604 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } attempt: 3
I20260808 04:47:41.520954 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.509350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.4.217.61:38513)
0808 04:47:41.509454 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.509458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.509470 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:47:41.509501 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.509504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.509539 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.509613 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.510750 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.511777 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.512755 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.513189 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.513290 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.515951 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.515956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.515962 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.515983 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.518157 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.519417 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.519421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.519422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.519513 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.519846 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.519853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.519858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.520086 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.520268 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.520668 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.520787 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:47:41.521142 26698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd: Bootstrap starting.
I20260808 04:47:41.521472 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.511290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38326)
0808 04:47:41.511374 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.511386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:41.511454 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:41.512332 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:41.512333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:41.512450 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:41.512501 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.512504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.513496 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.513500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.514350 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.514364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.515419 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.515424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.515517 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.516239 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:41.516252 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:41.517887 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:41.519911 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.519913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.519915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.520128 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.520393 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.520396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.520397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.520483 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.521013 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:41.521019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:41.521108 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 04:47:41.521209 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 04:47:41.521326 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 04:47:41.521822 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.511429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.4.217.60:35707)
0808 04:47:41.512022 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.512046 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.512060 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:47:41.512099 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.512103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.512148 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.512232 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.512611 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.513402 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.514497 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.514896 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.515008 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.518031 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.518035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.518037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.518061 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.519811 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.520220 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.520223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.520225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.520304 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.520612 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.520615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.520617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.520882 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.520978 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.521628 (+ 650us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.521708 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":113,"threads_started":1}
Aug 08 04:47:41 dist-test-slave-2kcd krb5kdc[26566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164461, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
I20260808 04:47:41.524181 26698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:41.524561 26970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933: Bootstrap starting.
I20260808 04:47:41.526302 26698 log.cc:824] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:41.527199 26970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:41.528092 26970 log.cc:824] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:41.528244 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:47:41.528578 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.518025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:35707 (local address 127.0.0.1:38338)
0808 04:47:41.518146 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:41.518160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:41.518237 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:41.518990 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:41.518993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:41.519174 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:41.519256 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.519262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.520203 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.520206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.521004 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:41.521015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.522460 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.522465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:41.522579 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.523023 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:41.523036 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:41.524408 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:41.526835 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.526839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.526841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.527090 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.527494 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:41.527500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:41.527502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:41.527605 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:41.528064 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:41.528067 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:41.528168 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 04:47:41.528258 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 04:47:41.528390 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:47:41.528967 26698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd: No bootstrap required, opened a new log
I20260808 04:47:41.529220 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.518580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.4.217.60:35707)
0808 04:47:41.518728 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.518732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.518747 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:47:41.518781 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.518783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.518826 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.519365 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.519439 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.520068 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.521374 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.522327 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.522779 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.524553 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.524557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.524559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.524583 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.526726 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.527220 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.527224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.527226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.527370 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.527741 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.527744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.527746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.527942 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.528900 (+ 958us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.528977 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.529107 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:47:41.532931 26970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933: No bootstrap required, opened a new log
I20260808 04:47:41.532948 26698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } }
I20260808 04:47:41.533193 26698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:41.533255 26698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731e707b6edc41dd96f90ebb7a5ec3fd, State: Initialized, Role: FOLLOWER
I20260808 04:47:41.533456 26834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c: Bootstrap starting.
I20260808 04:47:41.533784 26698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } }
I20260808 04:47:41.535154 26698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:47:41.535324 26625 tablet.cc:2369] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:47:41.536082 26834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:41.536996 26834 log.cc:824] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:41.537930 26970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } }
I20260808 04:47:41.538218 26970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:41.538295 26970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e53323f0f2b45e39af35eda94880933, State: Initialized, Role: FOLLOWER
I20260808 04:47:41.538738 26970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } }
I20260808 04:47:41.539239 26834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c: No bootstrap required, opened a new log
I20260808 04:47:41.539968 26970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:47:41.543740 26834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } }
I20260808 04:47:41.543926 26834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:41.543982 26834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e4d4db878574ee59621307d0859bc3c, State: Initialized, Role: FOLLOWER
I20260808 04:47:41.544389 26834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } }
I20260808 04:47:41.540169 26587 master_runner.cc:186] Error getting master registration for 127.4.217.61:38513: OK, instance_id { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" instance_seqno: 1786164461256105 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:47:41.545432 26587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:47:41.547340 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.530726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.4.217.60:35707)
0808 04:47:41.531242 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.531246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.533128 (+ 1882us) server_negotiation.cc:407] Connection header received
0808 04:47:41.533323 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.533327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.533369 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.533416 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.534404 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.535114 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.537419 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.537745 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.537825 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.539773 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.539776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.539779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.539803 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.544627 (+ 4824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.545923 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.545926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.545928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.546018 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.546382 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.546385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.546387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.546604 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.546723 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.547088 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.547199 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441}
W20260808 04:47:41.548411 27045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:47:41.548480 27045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:47:41.545846 26849 master_runner.cc:186] Error getting master registration for 127.4.217.61:38513: OK, instance_id { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" instance_seqno: 1786164461256105 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:47:41.549804 26849 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:47:41.550426 26834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:47:41.550791 27028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.550907 27028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:47:41.552960 27047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.553138 27047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:47:41.555406 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.530823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.4.217.61:38513)
0808 04:47:41.530946 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.530952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.531563 (+ 611us) server_negotiation.cc:407] Connection header received
0808 04:47:41.542393 (+ 10830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.542396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.542442 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.542489 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.543355 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.544048 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.545131 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.545443 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.545518 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.550699 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.550703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.550706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.550731 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.552913 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.553778 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.553782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.553784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.553872 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.554184 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.554188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.554190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.554388 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.554566 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.555139 (+ 573us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.555256 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:47:41.556399 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:41.530525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.4.217.62:36009)
0808 04:47:41.530794 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:41.530804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:41.536538 (+ 5734us) server_negotiation.cc:407] Connection header received
0808 04:47:41.536658 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:41.536661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:41.536706 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:41.536816 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:41.546800 (+ 9984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.547602 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.548726 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:41.549134 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:41.549212 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:41.552147 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:41.552150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:41.552153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:41.552179 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:41.554001 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.554518 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.554521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.554523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.554618 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:41.554930 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:41.554933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:41.554934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:41.555190 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:41.555316 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:41.556048 (+ 732us) server_negotiation.cc:299] Negotiation successful
0808 04:47:41.556265 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
W20260808 04:47:41.556479 27025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:47:41.556530 27025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:47:41.556712 27006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.556866 27006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [sys.catalog]: This master's current role is: FOLLOWER
W20260808 04:47:41.559765 27070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:47:41.559865 27070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:47:41.564548 26714 master_runner.cc:263] No leader master found from master 7e4d4db878574ee59621307d0859bc3c
I20260808 04:47:41.564635 26714 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:47:41.595806 27028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:47:41.596052 27028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } }
I20260808 04:47:41.597855 26660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e53323f0f2b45e39af35eda94880933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" is_pre_election: true
I20260808 04:47:41.598191 26660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e53323f0f2b45e39af35eda94880933 in term 0.
I20260808 04:47:41.598642 26891 leader_election.cc:302] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731e707b6edc41dd96f90ebb7a5ec3fd, 8e53323f0f2b45e39af35eda94880933; no voters:
I20260808 04:47:41.602092 26775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e53323f0f2b45e39af35eda94880933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e4d4db878574ee59621307d0859bc3c" is_pre_election: true
I20260808 04:47:41.602393 26775 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e53323f0f2b45e39af35eda94880933 in term 0.
I20260808 04:47:41.602774 27028 leader_election.cc:288] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 731e707b6edc41dd96f90ebb7a5ec3fd (127.4.217.62:36009), 7e4d4db878574ee59621307d0859bc3c (127.4.217.61:38513)
I20260808 04:47:41.603082 27028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:47:41.603178 27028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:47:41.603236 27028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:41.604600 27028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } }
I20260808 04:47:41.605250 26660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e53323f0f2b45e39af35eda94880933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd"
I20260808 04:47:41.605378 26660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:41.606642 26660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e53323f0f2b45e39af35eda94880933 in term 1.
I20260808 04:47:41.606957 26891 leader_election.cc:302] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731e707b6edc41dd96f90ebb7a5ec3fd, 8e53323f0f2b45e39af35eda94880933; no voters:
I20260808 04:47:41.607455 26775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e53323f0f2b45e39af35eda94880933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e4d4db878574ee59621307d0859bc3c"
I20260808 04:47:41.607583 26775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:41.609406 26775 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e53323f0f2b45e39af35eda94880933 in term 1.
I20260808 04:47:41.609773 27028 leader_election.cc:288] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [CANDIDATE]: Term 1 election: Requested vote from peers 731e707b6edc41dd96f90ebb7a5ec3fd (127.4.217.62:36009), 7e4d4db878574ee59621307d0859bc3c (127.4.217.61:38513)
I20260808 04:47:41.609921 27028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:47:41.610000 27028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [term 1 LEADER]: Becoming Leader. State: Replica: 8e53323f0f2b45e39af35eda94880933, State: Running, Role: LEADER
I20260808 04:47:41.610353 27028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } }
I20260808 04:47:41.615669 27028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e53323f0f2b45e39af35eda94880933. Latest consensus state: current_term: 1 leader_uuid: "8e53323f0f2b45e39af35eda94880933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.615782 27028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:41.616261 27076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:47:41.617290 27076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:47:41.626850 26775 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [term 1 FOLLOWER]: Refusing update from remote peer 8e53323f0f2b45e39af35eda94880933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:47:41.627506 27074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:47:41.631785 27047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e53323f0f2b45e39af35eda94880933. Latest consensus state: current_term: 1 leader_uuid: "8e53323f0f2b45e39af35eda94880933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.631922 27047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:47:41.632989 26660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [term 1 FOLLOWER]: Refusing update from remote peer 8e53323f0f2b45e39af35eda94880933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:47:41.633611 27074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:47:41.639007 27006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e53323f0f2b45e39af35eda94880933. Latest consensus state: current_term: 1 leader_uuid: "8e53323f0f2b45e39af35eda94880933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.639153 27006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:47:41.656522 27074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e53323f0f2b45e39af35eda94880933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.657573 27074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:41.658360 27006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e53323f0f2b45e39af35eda94880933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.658707 27006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:47:41.659828 27077 mvcc.cc:205] Tried to move back new op lower bound from 7316129634816274432 to 7316129634759667712. Current Snapshot: MvccSnapshot[applied={T|T < 7316129634816274432}]
I20260808 04:47:41.661060 27074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e53323f0f2b45e39af35eda94880933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.661177 27074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:41.662729 27047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e53323f0f2b45e39af35eda94880933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731e707b6edc41dd96f90ebb7a5ec3fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36009 } } peers { permanent_uuid: "7e4d4db878574ee59621307d0859bc3c" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38513 } } peers { permanent_uuid: "8e53323f0f2b45e39af35eda94880933" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 35707 } } }
I20260808 04:47:41.662873 27047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:47:41.663659 27081 mvcc.cc:205] Tried to move back new op lower bound from 7316129634816274432 to 7316129634759667712. Current Snapshot: MvccSnapshot[applied={T|T < 7316129634816274432 or (T in {7316129634816274432})}]
I20260808 04:47:41.663889 27078 mvcc.cc:205] Tried to move back new op lower bound from 7316129634816274432 to 7316129634759667712. Current Snapshot: MvccSnapshot[applied={T|T < 7316129634816274432}]
I20260808 04:47:41.665629 27076 catalog_manager.cc:1347] Generated new cluster ID: 78cfbe7e639746d2bdd431025bad773b
I20260808 04:47:41.665696 27076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:47:41.701632 27076 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:47:41.702986 27076 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:47:41.721570 27076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e53323f0f2b45e39af35eda94880933: Generated new TSK 0
I20260808 04:47:41.722466 27076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:47:42.557963 27025 catalog_manager.cc:1259] Loaded cluster ID: 78cfbe7e639746d2bdd431025bad773b
I20260808 04:47:42.558045 27025 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd: loading cluster ID for follower catalog manager: success
I20260808 04:47:42.559675 27025 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd: acquiring CA information for follower catalog manager: success
I20260808 04:47:42.560896 27025 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 731e707b6edc41dd96f90ebb7a5ec3fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:47:42.561044 27070 catalog_manager.cc:1259] Loaded cluster ID: 78cfbe7e639746d2bdd431025bad773b
I20260808 04:47:42.561132 27070 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c: loading cluster ID for follower catalog manager: success
I20260808 04:47:42.562822 27070 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c: acquiring CA information for follower catalog manager: success
I20260808 04:47:42.563889 27070 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e4d4db878574ee59621307d0859bc3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:47:42.823498 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 26587
I20260808 04:47:42.839322 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 26714
I20260808 04:47:42.854478 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 26849
2026-08-08T04:47:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3380 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:47:42 dist-test-slave-2kcd krb5kdc[27104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:47:42 dist-test-slave-2kcd krb5kdc[27104](info): set up 2 sockets
Aug 08 04:47:42 dist-test-slave-2kcd krb5kdc[27104](info): commencing operation
krb5kdc: starting...
W20260808 04:47:44.111025 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:47:44 dist-test-slave-2kcd krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:47:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:47:44.247792 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:40433
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:40433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:44.368583 27121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:44.368911 27121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:44.368993 27121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:44.373026 27121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:47:44.373123 27121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:44.373163 27121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:44.373224 27121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:47:44.373281 27121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:47:44.378700 27121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:40433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:40433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:44.379824 27121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:44.380926 27121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:44.389580 27131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:44.389595 27127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:44.389708 27121 server_base.cc:1034] running on GCE node
W20260808 04:47:44.389580 27129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:44.390479 27121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:44.391516 27121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:44.392696 27121 hybrid_clock.cc:648] HybridClock initialized: now 1786164464392675 us; error 30 us; skew 500 ppm
Aug 08 04:47:44 dist-test-slave-2kcd krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164464, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:44.395540 27121 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:47:44.396410 27121 webserver.cc:460] Webserver started at http://127.4.217.62:34939/ using document root <none> and password file <none>
I20260808 04:47:44.396992 27121 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:44.397063 27121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:44.397228 27121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:47:44.399245 27121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "48e4e0d226654a41aed7bc9706bed224"
format_stamp: "Formatted at 2026-08-08 04:47:44 on dist-test-slave-2kcd"
server_key: "de309e92541d43a0b76c2932a6de4959"
server_key_iv: "6df6914e2df6a521ace410ead357b11e"
server_key_version: "encryptionkey@0"
I20260808 04:47:44.399763 27121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "48e4e0d226654a41aed7bc9706bed224"
format_stamp: "Formatted at 2026-08-08 04:47:44 on dist-test-slave-2kcd"
server_key: "de309e92541d43a0b76c2932a6de4959"
server_key_iv: "6df6914e2df6a521ace410ead357b11e"
server_key_version: "encryptionkey@0"
I20260808 04:47:44.403475 27121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:47:44.405942 27145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:44.407164 27121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:47:44.407280 27121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "48e4e0d226654a41aed7bc9706bed224"
format_stamp: "Formatted at 2026-08-08 04:47:44 on dist-test-slave-2kcd"
server_key: "de309e92541d43a0b76c2932a6de4959"
server_key_iv: "6df6914e2df6a521ace410ead357b11e"
server_key_version: "encryptionkey@0"
I20260808 04:47:44.407390 27121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:44.417008 27121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:44.419695 27121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:44.419848 27121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:44.427254 27121 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:40433
I20260808 04:47:44.427284 27229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:40433 every 8 connection(s)
I20260808 04:47:44.428223 27121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:47:44.431084 27234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:44.434747 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 27121
I20260808 04:47:44.434857 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:47:44.435155 4964 external_mini_cluster.cc:1428] Setting key f41ab4b87e37698a9d4603188cf46373
I20260808 04:47:44.437228 27234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224: Bootstrap starting.
I20260808 04:47:44.440045 27234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:44.440980 27234 log.cc:824] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:44.443244 27234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224: No bootstrap required, opened a new log
Aug 08 04:47:44 dist-test-slave-2kcd krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:44.447402 27234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4e0d226654a41aed7bc9706bed224" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40433 } }
I20260808 04:47:44.447636 27234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:44.447701 27234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e4e0d226654a41aed7bc9706bed224, State: Initialized, Role: FOLLOWER
I20260808 04:47:44.448226 27234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4e0d226654a41aed7bc9706bed224" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40433 } }
I20260808 04:47:44.448417 27234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:47:44.448486 27234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:47:44.448619 27234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:44.449752 27234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4e0d226654a41aed7bc9706bed224" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40433 } }
I20260808 04:47:44.450147 27234 leader_election.cc:302] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48e4e0d226654a41aed7bc9706bed224; no voters:
I20260808 04:47:44.450482 27234 leader_election.cc:288] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:47:44.450588 27241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:47:44.450810 27241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [term 1 LEADER]: Becoming Leader. State: Replica: 48e4e0d226654a41aed7bc9706bed224, State: Running, Role: LEADER
I20260808 04:47:44.450980 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:44.436581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.4.217.62:40433)
0808 04:47:44.437031 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:44.437062 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:44.437108 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:47:44.437915 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:44.437938 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:44.438371 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:44.438756 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:44.439489 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:44.441054 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:44.442159 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:44.442820 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:44.442992 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:44.445819 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:44.445824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:44.445839 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:44.445866 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:44.447716 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:44.448270 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:44.448275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:44.448279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:44.448444 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:44.448837 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:44.448840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:44.448842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:44.449242 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:44.449403 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:44.449702 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 04:47:44.450002 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":143,"threads_started":1}
I20260808 04:47:44.451200 27241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4e0d226654a41aed7bc9706bed224" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40433 } }
I20260808 04:47:44.451771 27234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:47:44.452915 27243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48e4e0d226654a41aed7bc9706bed224. Latest consensus state: current_term: 1 leader_uuid: "48e4e0d226654a41aed7bc9706bed224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4e0d226654a41aed7bc9706bed224" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40433 } } }
I20260808 04:47:44.453037 27243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:44.453387 27242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48e4e0d226654a41aed7bc9706bed224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4e0d226654a41aed7bc9706bed224" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 40433 } } }
I20260808 04:47:44.453474 27242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:44.454084 27253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:47:44.454591 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:47:44.457281 27253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:47:44.463191 27253 catalog_manager.cc:1347] Generated new cluster ID: 1691ad8668044959b452bb3c43f71aa7
I20260808 04:47:44.463255 27253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:47:44.498584 27253 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:47:44.499922 27253 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:47:44.512357 27253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48e4e0d226654a41aed7bc9706bed224: Generated new TSK 0
I20260808 04:47:44.513149 27253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:47:44.562032 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 27121
2026-08-08T04:47:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1714 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:47:44 dist-test-slave-2kcd krb5kdc[27269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:47:44 dist-test-slave-2kcd krb5kdc[27269](info): set up 2 sockets
Aug 08 04:47:44 dist-test-slave-2kcd krb5kdc[27269](info): commencing operation
krb5kdc: starting...
W20260808 04:47:45.829300 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:47:45 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:47:45.954981 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:34733
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:34733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:46.067319 27293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:46.067620 27293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:46.067695 27293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:46.071761 27293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:47:46.071837 27293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:46.071877 27293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:46.071929 27293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:47:46.071971 27293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:47:46.077077 27293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:34733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:46.078089 27293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:46.079110 27293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:46.087610 27302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:46.087661 27306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:46.087649 27303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:46.087878 27293 server_base.cc:1034] running on GCE node
I20260808 04:47:46.088404 27293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:46.089650 27293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:46.090848 27293 hybrid_clock.cc:648] HybridClock initialized: now 1786164466090803 us; error 55 us; skew 500 ppm
Aug 08 04:47:46 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:46.093732 27293 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:47:46.094643 27293 webserver.cc:460] Webserver started at http://127.4.217.62:42929/ using document root <none> and password file <none>
I20260808 04:47:46.095276 27293 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:46.095342 27293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:46.095594 27293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:47:46.097676 27293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "426bb1182fa74334a9e89e654ad67fa4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "bae2c3ed3c6f49359c5045883174b9ed"
server_key_iv: "8cb14fed279c10ea49e64a0d9b1cee22"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.098318 27293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "426bb1182fa74334a9e89e654ad67fa4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "bae2c3ed3c6f49359c5045883174b9ed"
server_key_iv: "8cb14fed279c10ea49e64a0d9b1cee22"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.102103 27293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:47:46.104617 27320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.105870 27293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:47:46.106010 27293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "426bb1182fa74334a9e89e654ad67fa4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "bae2c3ed3c6f49359c5045883174b9ed"
server_key_iv: "8cb14fed279c10ea49e64a0d9b1cee22"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.106118 27293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:46.118394 27293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:46.121718 27293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:46.121919 27293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:46.129052 27293 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:34733
I20260808 04:47:46.129379 27419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:34733 every 8 connection(s)
I20260808 04:47:46.130095 27293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:47:46.130808 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 27293
I20260808 04:47:46.130954 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:47:46.131240 4964 external_mini_cluster.cc:1428] Setting key 90c8e9c71645631fb67a6fa21b5e93c7
I20260808 04:47:46.133343 27420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:46.140033 27420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Bootstrap starting.
Aug 08 04:47:46 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:46.142560 27420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:46.143314 27420 log.cc:824] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:46.145273 27420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: No bootstrap required, opened a new log
I20260808 04:47:46.145354 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.132567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.4.217.62:34733)
0808 04:47:46.133001 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.133016 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.133054 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:47:46.133788 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.133806 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.134122 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.134390 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:46.135285 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.136292 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.137463 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.137968 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.138083 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.140867 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:46.140873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:46.140885 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:46.140911 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:46.142563 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.143047 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.143057 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.143061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.143170 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.143461 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.143468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.143469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.143839 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:46.144053 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.144331 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.144616 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":292,"thread_start_us":162,"threads_started":1}
I20260808 04:47:46.148847 27420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:46.149087 27420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:46.149168 27420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426bb1182fa74334a9e89e654ad67fa4, State: Initialized, Role: FOLLOWER
I20260808 04:47:46.149582 27420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:46.149745 27420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:47:46.149816 27420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:47:46.149919 27420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:46.150878 27420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:46.151216 27420 leader_election.cc:302] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 426bb1182fa74334a9e89e654ad67fa4; no voters:
I20260808 04:47:46.151510 27420 leader_election.cc:288] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:47:46.151669 27433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:47:46.152009 27433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 1 LEADER]: Becoming Leader. State: Replica: 426bb1182fa74334a9e89e654ad67fa4, State: Running, Role: LEADER
I20260808 04:47:46.152298 27433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:46.152710 27420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:47:46.153874 27434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "426bb1182fa74334a9e89e654ad67fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } } }
I20260808 04:47:46.154026 27434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:46.154373 27435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 426bb1182fa74334a9e89e654ad67fa4. Latest consensus state: current_term: 1 leader_uuid: "426bb1182fa74334a9e89e654ad67fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } } }
I20260808 04:47:46.154444 27445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:47:46.155198 27435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:46.157579 27445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:47:46.163080 27445 catalog_manager.cc:1347] Generated new cluster ID: 444995ef7fa3404fb5ae45ead5fa480e
I20260808 04:47:46.163143 27445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:47:46.196972 27445 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:47:46.197829 27445 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:47:46.211145 27445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Generated new TSK 0
I20260808 04:47:46.212208 27445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:47:46.252063 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34733
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:46.384542 27461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:46.384773 27461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:46.384835 27461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:46.388536 27461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:46.388597 27461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:46.388652 27461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:47:46.393627 27461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:46.394537 27461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:46.395479 27461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:46.405285 27473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:46.405309 27477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:46.405290 27461 server_base.cc:1034] running on GCE node
W20260808 04:47:46.405278 27475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:46.405982 27461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:46.406648 27461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:46.407838 27461 hybrid_clock.cc:648] HybridClock initialized: now 1786164466407806 us; error 44 us; skew 500 ppm
Aug 08 04:47:46 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:46.410830 27461 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:47:46.411749 27461 webserver.cc:460] Webserver started at http://127.4.217.1:44217/ using document root <none> and password file <none>
I20260808 04:47:46.412279 27461 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:46.412336 27461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:46.412506 27461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:47:46.414582 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "6f8616ea813494868cb7a19ea7888ee7"
server_key_iv: "2e0f1cf26eba55d39cb46ba205ff632d"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.415096 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "6f8616ea813494868cb7a19ea7888ee7"
server_key_iv: "2e0f1cf26eba55d39cb46ba205ff632d"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.418733 27461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:47:46.421136 27487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.422396 27461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:47:46.422523 27461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "6f8616ea813494868cb7a19ea7888ee7"
server_key_iv: "2e0f1cf26eba55d39cb46ba205ff632d"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.422626 27461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:46.438125 27461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:46.440986 27461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:46.441184 27461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:46.441784 27461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:46.442615 27461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:46.442663 27461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.442734 27461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:46.442763 27461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.452389 27461 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:33689
I20260808 04:47:46.452417 27663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:33689 every 8 connection(s)
I20260808 04:47:46.453464 27461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:47:46.454299 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 27461
I20260808 04:47:46.454452 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:47:46.454762 4964 external_mini_cluster.cc:1428] Setting key 45ac3cc0ab1ebeaca69d8bb48da2a4cd
Aug 08 04:47:46 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:47:46.469374 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.455574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:46329 (local address 127.4.217.62:34733)
0808 04:47:46.455719 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.455724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.456799 (+ 1075us) server_negotiation.cc:407] Connection header received
0808 04:47:46.458761 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.458766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.458835 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.458969 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:46.460080 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.460651 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.462543 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.462781 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.462866 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.465771 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:46.465779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:46.465784 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:46.465820 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:46.467236 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.467748 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.467751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.467752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.467826 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.468141 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.468143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.468144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.468309 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:46.468394 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.468998 (+ 604us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.469133 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:47:46.470199 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.456026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:46329)
0808 04:47:46.456599 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:46.456656 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:46.458509 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:46.459109 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:46.459115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:46.459467 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:46.459932 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.459945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.460818 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.460824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.462416 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.462431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.462949 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.462962 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.463199 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.463852 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:46.463875 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:46.465594 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:46.467344 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:46.467348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:46.467360 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:46.467664 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:46.467923 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:46.467925 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:46.467926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:46.468054 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:46.468394 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:46.468399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:46.468712 (+ 313us) client_negotiation.cc:769] Sending connection context
0808 04:47:46.468941 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 04:47:46.469293 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":136,"threads_started":1}
I20260808 04:47:46.471592 27665 heartbeater.cc:347] Connected to a master server at 127.4.217.62:34733
I20260808 04:47:46.471843 27665 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:46.472442 27665 heartbeater.cc:511] Master 127.4.217.62:34733 requested a full tablet report, sending...
I20260808 04:47:46.474218 27353 ts_manager.cc:194] Registered new tserver with Master: 6cbe985c1e4f4d898bc68b0f2c2bd8fc (127.4.217.1:33689)
I20260808 04:47:46.475423 27353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:46329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:47:46.485204 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34733
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:46.593262 27680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:46.593513 27680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:46.593585 27680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:46.597486 27680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:46.597555 27680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:46.597638 27680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:47:46.602615 27680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:46.603559 27680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:46.604503 27680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:46.613951 27690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:46.614022 27688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:46.614074 27696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:46.614197 27680 server_base.cc:1034] running on GCE node
I20260808 04:47:46.614681 27680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:46.615286 27680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:46.616451 27680 hybrid_clock.cc:648] HybridClock initialized: now 1786164466616426 us; error 32 us; skew 500 ppm
Aug 08 04:47:46 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:46.619258 27680 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:47:46.620014 27680 webserver.cc:460] Webserver started at http://127.4.217.2:34899/ using document root <none> and password file <none>
I20260808 04:47:46.620549 27680 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:46.620606 27680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:46.620756 27680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:47:46.622771 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "a0ac27da05a34d88be7e4e25077765ac"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "81c6e74631c28239bf1eae2d7e8a2c1d"
server_key_iv: "27e10afb66c5f96682115c52dd1eade9"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.623318 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "a0ac27da05a34d88be7e4e25077765ac"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "81c6e74631c28239bf1eae2d7e8a2c1d"
server_key_iv: "27e10afb66c5f96682115c52dd1eade9"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.627132 27680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 04:47:46.629408 27711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.630540 27680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:47:46.630633 27680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "a0ac27da05a34d88be7e4e25077765ac"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "81c6e74631c28239bf1eae2d7e8a2c1d"
server_key_iv: "27e10afb66c5f96682115c52dd1eade9"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.630710 27680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:46.644761 27680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:46.647922 27680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:46.648077 27680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:46.648698 27680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:46.649540 27680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:46.649590 27680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.649655 27680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:46.649693 27680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.659168 27680 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:37195
I20260808 04:47:46.659198 27898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:37195 every 8 connection(s)
I20260808 04:47:46.659988 27680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:47:46.661263 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 27680
I20260808 04:47:46.661391 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:47:46.661717 4964 external_mini_cluster.cc:1428] Setting key abeccd6c1be8a8139534840754a00637
Aug 08 04:47:46 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:47:46.674471 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.661870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:57919 (local address 127.4.217.62:34733)
0808 04:47:46.662004 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.662007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.663063 (+ 1056us) server_negotiation.cc:407] Connection header received
0808 04:47:46.664386 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.664389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.664442 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.664520 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:46.665704 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.666233 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.667600 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.667924 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.668014 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.670877 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:46.670880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:46.670882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:46.670905 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:46.672343 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.672898 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.672902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.672916 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.672996 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.673343 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.673346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.673348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.673517 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:46.673631 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.674186 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.674308 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:47:46.675204 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.662399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:57919)
0808 04:47:46.662909 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:46.662959 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:46.664185 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:46.664676 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:46.664682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:46.665075 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:46.665563 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.665575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.666367 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.666370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.667453 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.667466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.668096 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.668102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.668316 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.668970 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:46.668996 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:46.670689 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:46.672478 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:46.672484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:46.672497 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:46.672752 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:46.673095 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:46.673098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:46.673100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:46.673239 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:46.673632 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:46.673639 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:46.673943 (+ 304us) client_negotiation.cc:769] Sending connection context
0808 04:47:46.674124 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 04:47:46.674414 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":151,"threads_started":1}
I20260808 04:47:46.676453 27903 heartbeater.cc:347] Connected to a master server at 127.4.217.62:34733
I20260808 04:47:46.676688 27903 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
I20260808 04:47:46.677230 27903 heartbeater.cc:511] Master 127.4.217.62:34733 requested a full tablet report, sending...
I20260808 04:47:46.678380 27353 ts_manager.cc:194] Registered new tserver with Master: a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195)
I20260808 04:47:46.678952 27353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:57919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:47:46.691468 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34733
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:46.803637 27912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:46.803889 27912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:46.803929 27912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:46.807827 27912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:46.807876 27912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:46.807930 27912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:47:46.813321 27912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:46.814245 27912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:46.815169 27912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:46.824635 27926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:46.825016 27920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:46.825004 27912 server_base.cc:1034] running on GCE node
W20260808 04:47:46.825249 27924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:46.825644 27912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:46.826340 27912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:46.827534 27912 hybrid_clock.cc:648] HybridClock initialized: now 1786164466827499 us; error 45 us; skew 500 ppm
Aug 08 04:47:46 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:46.832743 27912 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:47:46.833633 27912 webserver.cc:460] Webserver started at http://127.4.217.3:41985/ using document root <none> and password file <none>
I20260808 04:47:46.834218 27912 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:46.834334 27912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:46.834513 27912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:47:46.836439 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "26d000fd1bad4f9a88294b269e2bf1e4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "2e3d7489ef02f0250a3b3906074da947"
server_key_iv: "dc6b3a2f7ca72d1c0541703e4cee6430"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.837028 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "26d000fd1bad4f9a88294b269e2bf1e4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "2e3d7489ef02f0250a3b3906074da947"
server_key_iv: "dc6b3a2f7ca72d1c0541703e4cee6430"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.840515 27912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:47:46.842893 27938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.844046 27912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:47:46.844185 27912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "26d000fd1bad4f9a88294b269e2bf1e4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "2e3d7489ef02f0250a3b3906074da947"
server_key_iv: "dc6b3a2f7ca72d1c0541703e4cee6430"
server_key_version: "encryptionkey@0"
I20260808 04:47:46.844282 27912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:46.867401 27912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:46.870493 27912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:46.870718 27912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:46.871380 27912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:46.872285 27912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:46.872337 27912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.872428 27912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:46.872473 27912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:46.882578 27912 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:44435
I20260808 04:47:46.882609 28129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:44435 every 8 connection(s)
I20260808 04:47:46.883479 27912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:47:46.887777 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 27912
I20260808 04:47:46.887938 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:47:46.888278 4964 external_mini_cluster.cc:1428] Setting key 04175ea3c528da0f2011132c2d67836d
Aug 08 04:47:46 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:46.897505 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.885390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:50011 (local address 127.4.217.62:34733)
0808 04:47:46.885545 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.885550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.886431 (+ 881us) server_negotiation.cc:407] Connection header received
0808 04:47:46.887709 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.887712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.887760 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.887821 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:46.889522 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.890135 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.891682 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.891889 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.891972 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.894347 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:46.894350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:46.894352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:46.894377 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:46.895665 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.896159 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.896161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.896163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.896238 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.896537 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.896540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.896548 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.896709 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:46.896833 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.897282 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.897388 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:47:46.898241 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.885808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:50011)
0808 04:47:46.886278 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:46.886320 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:46.887429 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:46.888065 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:46.888075 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:46.888578 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:46.889337 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.889357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.890386 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.890390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.891564 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.891575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.892023 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.892026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.892160 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.892643 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:46.892658 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:46.894192 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:46.895778 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:46.895782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:46.895793 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:46.896066 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:46.896319 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:46.896321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:46.896322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:46.896458 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:46.896846 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:46.896851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:46.897076 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 04:47:46.897225 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:47:46.897458 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260808 04:47:46.899272 28130 heartbeater.cc:347] Connected to a master server at 127.4.217.62:34733
I20260808 04:47:46.899529 28130 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:46.900000 28130 heartbeater.cc:511] Master 127.4.217.62:34733 requested a full tablet report, sending...
I20260808 04:47:46.901057 27353 ts_manager.cc:194] Registered new tserver with Master: 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435)
I20260808 04:47:46.901693 27353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:50011
I20260808 04:47:46.902477 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:47:46.911959 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.904425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.4.217.62:34733)
0808 04:47:46.904573 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.904578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.904812 (+ 234us) server_negotiation.cc:407] Connection header received
0808 04:47:46.904953 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.904956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.904997 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.905087 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:46.905778 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.906287 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.907123 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.907350 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.907462 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.908839 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:46.908842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:46.908845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:46.908869 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:46.910263 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.910713 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.910717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.910718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.910797 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:46.911110 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:46.911113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:46.911115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:46.911311 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:46.911452 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.911691 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.911802 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:47:46.914595 27353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:47:46.917560 27353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:47:46.931869 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.925879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33689 (local address 127.0.0.1:59616)
0808 04:47:46.926625 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:46.926685 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:46.926840 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:46.927626 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:46.927636 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:46.927663 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:46.927780 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.927787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.929053 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.929059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.930277 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.930288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.931273 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.931277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.931405 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.931454 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 04:47:46.931582 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 04:47:46.931712 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260808 04:47:46.931882 28150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.926099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.4.217.1:33689)
0808 04:47:46.926670 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.926687 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.926781 (+ 94us) server_negotiation.cc:407] Connection header received
0808 04:47:46.927065 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.927071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.927281 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.927435 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:46.927909 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.928927 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.930419 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.931146 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.931254 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.931394 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.931625 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.931725 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":100,"threads_started":1}
I20260808 04:47:46.932420 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.926100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:44435 (local address 127.0.0.1:57866)
0808 04:47:46.927252 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:46.927269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:46.927391 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:46.927918 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:46.927922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:46.927942 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:46.928084 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.928094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.929545 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.929549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.930636 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.930647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.931924 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.931930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.932070 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.932102 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:47:46.932185 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:47:46.932281 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"thread_start_us":127,"threads_started":1}
I20260808 04:47:46.932931 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.926685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57866 (local address 127.4.217.3:44435)
0808 04:47:46.927052 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.927066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.927503 (+ 437us) server_negotiation.cc:407] Connection header received
0808 04:47:46.927590 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.927595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.927781 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.927942 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:46.928218 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.929381 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.930749 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.931595 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.931700 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.931848 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.932622 (+ 774us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.932737 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":169,"threads_started":1}
I20260808 04:47:46.932967 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.926321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:37195 (local address 127.0.0.1:47998)
0808 04:47:46.927478 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:46.927493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:46.927595 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:46.928254 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:46.928257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:46.928275 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:46.928371 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.928381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.929784 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.929789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.930986 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.930999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.932269 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.932274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.932380 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.932406 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:47:46.932733 (+ 327us) client_negotiation.cc:240] Negotiation successful
0808 04:47:46.932822 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":25,"thread_start_us":85,"threads_started":1}
I20260808 04:47:46.935148 27563 tablet_service.cc:1467] Processing CreateTablet for tablet f85f41c74cc847eba4825778913b3f81 (DEFAULT_TABLE table=test-table [id=ebb4cb39ea8a4f47b932272b5e36a44e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:47:46.935708 28019 tablet_service.cc:1467] Processing CreateTablet for tablet f85f41c74cc847eba4825778913b3f81 (DEFAULT_TABLE table=test-table [id=ebb4cb39ea8a4f47b932272b5e36a44e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:47:46.936332 27563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85f41c74cc847eba4825778913b3f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:46.936736 28019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85f41c74cc847eba4825778913b3f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:46.940738 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.926803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.4.217.2:37195)
0808 04:47:46.927350 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.927365 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.927689 (+ 324us) server_negotiation.cc:407] Connection header received
0808 04:47:46.927768 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.927773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.927951 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.928090 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:46.928512 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.929646 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.931137 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.931896 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.932259 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.932403 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.932814 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.932924 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":137,"threads_started":1}
I20260808 04:47:46.943024 27787 tablet_service.cc:1467] Processing CreateTablet for tablet f85f41c74cc847eba4825778913b3f81 (DEFAULT_TABLE table=test-table [id=ebb4cb39ea8a4f47b932272b5e36a44e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:47:46.943886 28160 tablet_bootstrap.cc:492] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Bootstrap starting.
I20260808 04:47:46.943970 27787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85f41c74cc847eba4825778913b3f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:46.944360 28159 tablet_bootstrap.cc:492] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: Bootstrap starting.
I20260808 04:47:46.947396 28159 tablet_bootstrap.cc:654] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:46.948378 28159 log.cc:824] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:46.950203 28159 tablet_bootstrap.cc:492] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: No bootstrap required, opened a new log
I20260808 04:47:46.950414 28159 ts_tablet_manager.cc:1397] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260808 04:47:46.951087 28160 tablet_bootstrap.cc:654] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:46.952227 28166 tablet_bootstrap.cc:492] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: Bootstrap starting.
I20260808 04:47:46.952355 28160 log.cc:824] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:46.954504 28160 tablet_bootstrap.cc:492] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: No bootstrap required, opened a new log
I20260808 04:47:46.954730 28160 ts_tablet_manager.cc:1397] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260808 04:47:46.954845 28166 tablet_bootstrap.cc:654] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:46.954690 28159 raft_consensus.cc:348] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:46.955014 28159 raft_consensus.cc:374] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:46.955084 28159 raft_consensus.cc:729] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d000fd1bad4f9a88294b269e2bf1e4, State: Initialized, Role: FOLLOWER
I20260808 04:47:46.955528 28159 consensus_queue.cc:260] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:46.955694 28166 log.cc:824] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:46.956521 28130 heartbeater.cc:503] Master 127.4.217.62:34733 was elected leader, sending a full tablet report...
I20260808 04:47:46.956528 28159 ts_tablet_manager.cc:1428] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260808 04:47:46.957885 28166 tablet_bootstrap.cc:492] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: No bootstrap required, opened a new log
I20260808 04:47:46.958076 28166 ts_tablet_manager.cc:1397] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:47:46.958776 28160 raft_consensus.cc:348] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:46.959052 28160 raft_consensus.cc:374] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:46.959107 28160 raft_consensus.cc:729] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cbe985c1e4f4d898bc68b0f2c2bd8fc, State: Initialized, Role: FOLLOWER
I20260808 04:47:46.959545 28160 consensus_queue.cc:260] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:46.960240 27665 heartbeater.cc:503] Master 127.4.217.62:34733 was elected leader, sending a full tablet report...
I20260808 04:47:46.960530 28160 ts_tablet_manager.cc:1428] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:47:46.961527 28166 raft_consensus.cc:348] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:46.961732 28166 raft_consensus.cc:374] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:46.961764 28166 raft_consensus.cc:729] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ac27da05a34d88be7e4e25077765ac, State: Initialized, Role: FOLLOWER
I20260808 04:47:46.962178 28166 consensus_queue.cc:260] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:46.962849 27903 heartbeater.cc:503] Master 127.4.217.62:34733 was elected leader, sending a full tablet report...
I20260808 04:47:46.963076 28166 ts_tablet_manager.cc:1428] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260808 04:47:46.967756 28168 raft_consensus.cc:482] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:47:46.967906 28168 raft_consensus.cc:504] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:46.968928 28168 leader_election.cc:288] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cbe985c1e4f4d898bc68b0f2c2bd8fc (127.4.217.1:33689), a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195)
I20260808 04:47:46.972827 28150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.969137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:36545 (local address 127.4.217.1:33689)
0808 04:47:46.969300 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.969306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.969321 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:47:46.969454 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.969457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.969546 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.969628 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:46.969893 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.970814 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.971852 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.972247 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.972360 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.972418 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.972639 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.972700 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:47:46.972833 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.969067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33689 (local address 127.4.217.3:36545)
0808 04:47:46.969229 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:46.969246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:46.969350 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:46.969654 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:46.969657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:46.969676 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:46.969792 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.969800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.970927 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.970932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.971723 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.971733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.972373 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.972377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.972511 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.972529 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:47:46.972622 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:47:46.972697 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1}
I20260808 04:47:46.973862 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.969400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:42745 (local address 127.4.217.2:37195)
0808 04:47:46.969534 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.969538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.969553 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:47:46.970762 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.970765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.970815 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.970905 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:46.971193 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.971685 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.972697 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.973314 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.973436 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.973493 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.973699 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.973763 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:47:46.973891 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.969169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:37195 (local address 127.4.217.3:42745)
0808 04:47:46.969471 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:46.969484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:46.969613 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:46.970915 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:46.970918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:46.970935 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:46.971081 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.971086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.971799 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.971803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.972573 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:46.972582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.973456 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.973460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:46.973574 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.973610 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:47:46.973683 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 04:47:46.973749 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260808 04:47:46.973990 27595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85f41c74cc847eba4825778913b3f81" candidate_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" is_pre_election: true
I20260808 04:47:46.974321 27595 raft_consensus.cc:2454] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d000fd1bad4f9a88294b269e2bf1e4 in term 0.
I20260808 04:47:46.974655 27823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85f41c74cc847eba4825778913b3f81" candidate_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ac27da05a34d88be7e4e25077765ac" is_pre_election: true
I20260808 04:47:46.974830 27946 leader_election.cc:302] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d000fd1bad4f9a88294b269e2bf1e4, 6cbe985c1e4f4d898bc68b0f2c2bd8fc; no voters:
I20260808 04:47:46.974893 27823 raft_consensus.cc:2454] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d000fd1bad4f9a88294b269e2bf1e4 in term 0.
I20260808 04:47:46.975106 28168 raft_consensus.cc:2781] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:47:46.975207 28168 raft_consensus.cc:482] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:47:46.975279 28168 raft_consensus.cc:3037] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:46.976202 28168 raft_consensus.cc:504] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:46.976585 28168 leader_election.cc:288] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [CANDIDATE]: Term 1 election: Requested vote from peers 6cbe985c1e4f4d898bc68b0f2c2bd8fc (127.4.217.1:33689), a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195)
I20260808 04:47:46.976859 27595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85f41c74cc847eba4825778913b3f81" candidate_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc"
I20260808 04:47:46.976946 27823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85f41c74cc847eba4825778913b3f81" candidate_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ac27da05a34d88be7e4e25077765ac"
I20260808 04:47:46.976992 27595 raft_consensus.cc:3037] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:46.977058 27823 raft_consensus.cc:3037] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:46.977944 27595 raft_consensus.cc:2454] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d000fd1bad4f9a88294b269e2bf1e4 in term 1.
I20260808 04:47:46.978010 27823 raft_consensus.cc:2454] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d000fd1bad4f9a88294b269e2bf1e4 in term 1.
I20260808 04:47:46.978263 27946 leader_election.cc:302] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d000fd1bad4f9a88294b269e2bf1e4, 6cbe985c1e4f4d898bc68b0f2c2bd8fc; no voters:
I20260808 04:47:46.978422 28168 raft_consensus.cc:2781] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:47:46.978669 28168 raft_consensus.cc:686] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 LEADER]: Becoming Leader. State: Replica: 26d000fd1bad4f9a88294b269e2bf1e4, State: Running, Role: LEADER
I20260808 04:47:46.978958 28168 consensus_queue.cc:237] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:46.981789 27353 catalog_manager.cc:5495] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3). New cstate: current_term: 1 leader_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } health_report { overall_health: UNKNOWN } } }
I20260808 04:47:46.996685 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:46.993566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.4.217.3:44435)
0808 04:47:46.993718 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:46.993721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:46.993764 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:47:46.993887 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:46.993889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:46.993953 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:46.994037 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:47:46.994240 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.994743 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.995538 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:46.995731 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:46.995802 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:46.996051 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:47:46.996401 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:47:46.996489 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:46.996550 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 04:47:46.996593 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:47:47.001875 27823 raft_consensus.cc:1261] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [term 1 FOLLOWER]: Refusing update from remote peer 26d000fd1bad4f9a88294b269e2bf1e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:47:47.001904 27595 raft_consensus.cc:1261] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Refusing update from remote peer 26d000fd1bad4f9a88294b269e2bf1e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:47:47.002591 28179 consensus_queue.cc:1035] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:47:47.002758 28168 consensus_queue.cc:1035] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:47:47.012197 28188 mvcc.cc:205] Tried to move back new op lower bound from 7316129656833290240 to 7316129656748085248. Current Snapshot: MvccSnapshot[applied={T|T < 7316129656833290240}]
I20260808 04:47:47.017928 27353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:47:47.018103 27353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:47:47.020509 27353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Sending DeleteTablet for 3 replicas of tablet f85f41c74cc847eba4825778913b3f81
I20260808 04:47:47.021366 28019 tablet_service.cc:1514] Processing DeleteTablet for tablet f85f41c74cc847eba4825778913b3f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:47 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:57866
I20260808 04:47:47.021397 27787 tablet_service.cc:1514] Processing DeleteTablet for tablet f85f41c74cc847eba4825778913b3f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:47 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:47998
I20260808 04:47:47.021643 27563 tablet_service.cc:1514] Processing DeleteTablet for tablet f85f41c74cc847eba4825778913b3f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:47 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:59616
I20260808 04:47:47.021829 28203 tablet_replica.cc:330] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: stopping tablet replica
I20260808 04:47:47.021968 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 27461
I20260808 04:47:47.027225 28204 tablet_replica.cc:330] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: stopping tablet replica
I20260808 04:47:47.027515 28204 raft_consensus.cc:2229] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:47:47.027829 28204 raft_consensus.cc:2258] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:47:47.030308 28203 raft_consensus.cc:2229] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:47:47.030603 28203 raft_consensus.cc:2258] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:47:47.033439 28203 ts_tablet_manager.cc:1905] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:47:47.034011 28204 ts_tablet_manager.cc:1905] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:47:47.035930 28203 ts_tablet_manager.cc:1918] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:47:47.036101 28203 log.cc:1196] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/f85f41c74cc847eba4825778913b3f81
I20260808 04:47:47.036410 28203 ts_tablet_manager.cc:1939] T f85f41c74cc847eba4825778913b3f81 P a0ac27da05a34d88be7e4e25077765ac: Deleting consensus metadata
I20260808 04:47:47.036640 28204 ts_tablet_manager.cc:1918] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:47:47.036753 28204 log.cc:1196] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/f85f41c74cc847eba4825778913b3f81
I20260808 04:47:47.037114 28204 ts_tablet_manager.cc:1939] T f85f41c74cc847eba4825778913b3f81 P 26d000fd1bad4f9a88294b269e2bf1e4: Deleting consensus metadata
I20260808 04:47:47.037798 27330 catalog_manager.cc:4841] TS a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195): tablet f85f41c74cc847eba4825778913b3f81 (table test-table [id=ebb4cb39ea8a4f47b932272b5e36a44e]) successfully deleted
I20260808 04:47:47.038153 27325 catalog_manager.cc:4841] TS 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435): tablet f85f41c74cc847eba4825778913b3f81 (table test-table [id=ebb4cb39ea8a4f47b932272b5e36a44e]) successfully deleted
W20260808 04:47:47.043586 27331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:33689 (error 108)
I20260808 04:47:47.043823 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 27680
I20260808 04:47:47.053428 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.053089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33689 (local address 127.0.0.1:59624)
0808 04:47:47.053236 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:47.053321 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:47:47.053673 27331 catalog_manager.cc:4584] TS 6cbe985c1e4f4d898bc68b0f2c2bd8fc (127.4.217.1:33689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f85f41c74cc847eba4825778913b3f81: Network error: Client connection negotiation failed: client connection to 127.4.217.1:33689: connect: Connection refused (error 111)
I20260808 04:47:47.058750 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 27912
I20260808 04:47:47.069150 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 27293
I20260808 04:47:47.081797 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:34733
--webserver_interface=127.4.217.62
--webserver_port=42929
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:34733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:47.193043 28208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:47.193337 28208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:47.193413 28208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:47.197347 28208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:47:47.197422 28208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:47.197458 28208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:47.197535 28208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:47:47.197570 28208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:47:47.202689 28208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:34733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=42929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:47.203727 28208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:47.204699 28208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:47.220584 28217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:47.220551 28221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:47.220604 28219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:47.221087 28208 server_base.cc:1034] running on GCE node
I20260808 04:47:47.221645 28208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:47.222728 28208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:47.223939 28208 hybrid_clock.cc:648] HybridClock initialized: now 1786164467223901 us; error 54 us; skew 500 ppm
Aug 08 04:47:47 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:47.226707 28208 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:47:47.227558 28208 webserver.cc:460] Webserver started at http://127.4.217.62:42929/ using document root <none> and password file <none>
I20260808 04:47:47.228080 28208 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:47.228152 28208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:47.231827 28208 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:47:47.233898 28233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:47.235069 28208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:47:47.235199 28208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "426bb1182fa74334a9e89e654ad67fa4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "bae2c3ed3c6f49359c5045883174b9ed"
server_key_iv: "8cb14fed279c10ea49e64a0d9b1cee22"
server_key_version: "encryptionkey@0"
I20260808 04:47:47.235761 28208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:47.276685 28208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:47.279753 28208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:47.279981 28208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:47.287031 28208 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:34733
I20260808 04:47:47.287340 28319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:34733 every 8 connection(s)
I20260808 04:47:47.288105 28208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:47:47.288969 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 28208
I20260808 04:47:47.289480 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:33689
--local_ip_for_outbound_sockets=127.4.217.1
--tserver_master_addrs=127.4.217.62:34733
--webserver_port=44217
--webserver_interface=127.4.217.1
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:47:47.293713 28322 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:47:47.295439 28322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Bootstrap starting.
I20260808 04:47:47.312702 28322 log.cc:824] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:47.321399 28322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:47:47.321766 28322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Bootstrap complete.
I20260808 04:47:47.325927 28322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:47.326305 28322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426bb1182fa74334a9e89e654ad67fa4, State: Initialized, Role: FOLLOWER
I20260808 04:47:47.326774 28322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:47.326885 28322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:47:47.326973 28322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:47:47.327103 28322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:47:47.328955 28322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:47.329286 28322 leader_election.cc:302] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 426bb1182fa74334a9e89e654ad67fa4; no voters:
I20260808 04:47:47.329612 28322 leader_election.cc:288] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:47:47.329718 28326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:47:47.329950 28326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 2 LEADER]: Becoming Leader. State: Replica: 426bb1182fa74334a9e89e654ad67fa4, State: Running, Role: LEADER
I20260808 04:47:47.330233 28326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:47.330824 28322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:47:47.331162 28326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 426bb1182fa74334a9e89e654ad67fa4. Latest consensus state: current_term: 2 leader_uuid: "426bb1182fa74334a9e89e654ad67fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } } }
I20260808 04:47:47.331284 28326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:47.331652 28331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:47:47.331683 28328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "426bb1182fa74334a9e89e654ad67fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } } }
I20260808 04:47:47.331904 28328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:47.337738 28331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:47:47.338483 28331 catalog_manager.cc:1259] Loaded cluster ID: 444995ef7fa3404fb5ae45ead5fa480e
I20260808 04:47:47.338547 28331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:47:47.339982 28331 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 04:47:47.344878 28331 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 04:47:47.349908 28331 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:47:47.350598 28331 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Loaded TSK: 0
I20260808 04:47:47.351747 28331 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:47:47.405390 28324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:47.405634 28324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:47.405710 28324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:47.409675 28324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:47.409747 28324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:47.409840 28324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:47:47.414875 28324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:33689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=44217
--tserver_master_addrs=127.4.217.62:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:47.415786 28324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:47.416920 28324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:47.427345 28378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:47.427321 28371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:47.427327 28370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:47.427954 28324 server_base.cc:1034] running on GCE node
I20260808 04:47:47.428310 28324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:47.429056 28324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:47.430235 28324 hybrid_clock.cc:648] HybridClock initialized: now 1786164467430214 us; error 38 us; skew 500 ppm
Aug 08 04:47:47 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:47.433480 28324 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:47:47.434391 28324 webserver.cc:460] Webserver started at http://127.4.217.1:44217/ using document root <none> and password file <none>
I20260808 04:47:47.434959 28324 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:47.435032 28324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:47.438952 28324 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:47:47.441316 28387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:47.442538 28324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:47:47.442631 28324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "6f8616ea813494868cb7a19ea7888ee7"
server_key_iv: "2e0f1cf26eba55d39cb46ba205ff632d"
server_key_version: "encryptionkey@0"
I20260808 04:47:47.443166 28324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:47.480623 28324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:47.483986 28324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:47.484146 28324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:47.484673 28324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:47.485798 28397 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:47:47.489897 28324 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:47:47.489974 28324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260808 04:47:47.490025 28324 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:47:47.491647 28324 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:47:47.491693 28324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 04:47:47.491973 28397 tablet_bootstrap.cc:492] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Bootstrap starting.
I20260808 04:47:47.502475 28324 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:33689
I20260808 04:47:47.502532 28598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:33689 every 8 connection(s)
I20260808 04:47:47.503494 28324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:47:47.507139 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 28324
I20260808 04:47:47.507552 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:37195
--local_ip_for_outbound_sockets=127.4.217.2
--tserver_master_addrs=127.4.217.62:34733
--webserver_port=34899
--webserver_interface=127.4.217.2
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:47:47.512109 28397 log.cc:824] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:47:47 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:47.520381 28397 tablet_bootstrap.cc:492] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:47:47.520828 28397 tablet_bootstrap.cc:492] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Bootstrap complete.
I20260808 04:47:47.522007 28397 ts_tablet_manager.cc:1397] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20260808 04:47:47.523412 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.505723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:47833 (local address 127.4.217.62:34733)
0808 04:47:47.506206 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:47.506225 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:47.506341 (+ 116us) server_negotiation.cc:407] Connection header received
0808 04:47:47.508088 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:47.508104 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:47.508492 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:47.508916 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:47.510472 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.511693 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.513134 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.513833 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.513987 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.517406 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:47.517413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:47.517429 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:47.517464 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:47.519599 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.520329 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.520334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.520340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.520495 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.520949 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.520953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.520954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.521390 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:47.521509 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:47.522315 (+ 806us) server_negotiation.cc:299] Negotiation successful
0808 04:47:47.522601 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":120,"threads_started":1}
I20260808 04:47:47.523437 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.505579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:47833)
0808 04:47:47.506157 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:47.506215 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:47.507410 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:47.508994 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:47.509003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:47.509547 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:47.510265 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.510284 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.511864 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.511867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.512982 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.512997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.514225 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.514229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.514405 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.515117 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:47.515137 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:47.517170 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:47.519758 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:47.519767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:47.519781 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:47.520171 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:47.520608 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:47.520611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:47.520613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:47.520824 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:47.521600 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:47.521606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:47.521891 (+ 285us) client_negotiation.cc:769] Sending connection context
0808 04:47:47.522074 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 04:47:47.522398 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":148,"threads_started":1}
I20260808 04:47:47.524745 28397 raft_consensus.cc:348] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:47.525350 28397 raft_consensus.cc:729] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cbe985c1e4f4d898bc68b0f2c2bd8fc, State: Initialized, Role: FOLLOWER
I20260808 04:47:47.525835 28397 consensus_queue.cc:260] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } }
I20260808 04:47:47.526371 28599 heartbeater.cc:347] Connected to a master server at 127.4.217.62:34733
I20260808 04:47:47.526616 28599 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:47.526861 28397 ts_tablet_manager.cc:1428] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:47:47.527150 28599 heartbeater.cc:511] Master 127.4.217.62:34733 requested a full tablet report, sending...
I20260808 04:47:47.529717 28257 ts_manager.cc:194] Registered new tserver with Master: 6cbe985c1e4f4d898bc68b0f2c2bd8fc (127.4.217.1:33689)
I20260808 04:47:47.533397 28257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:47833
I20260808 04:47:47.536187 28599 heartbeater.cc:503] Master 127.4.217.62:34733 was elected leader, sending a full tablet report...
Aug 08 04:47:47 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:47:47.543376 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.532719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33689 (local address 127.0.0.1:59632)
0808 04:47:47.533143 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:47.533188 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:47.533311 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:47.534082 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:47.534091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:47.534563 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:47.534667 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.534674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.535777 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.535780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.536486 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.536497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.537288 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.537292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.537402 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.538021 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:47.538033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:47.539645 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:47.541719 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:47.541727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:47.541731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:47.541964 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:47.542281 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:47.542283 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:47.542285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:47.542358 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:47.542880 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:47.542885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:47.542993 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 04:47:47.543102 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:47:47.543246 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":139,"threads_started":1}
I20260808 04:47:47.543658 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.532825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.4.217.1:33689)
0808 04:47:47.533611 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:47.533622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:47.533646 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:47:47.533719 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:47.533724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:47.533917 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:47.534055 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:47.534792 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.535631 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.536628 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.537153 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.537266 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.539780 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:47.539790 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:47.539803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:47.539830 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:47.541614 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.542078 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.542085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.542087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.542185 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.542447 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.542451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.542452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.542762 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:47.542894 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:47.543217 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 04:47:47.543390 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":673,"thread_start_us":110,"threads_started":1}
I20260808 04:47:47.544553 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet f85f41c74cc847eba4825778913b3f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:47 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:59632
I20260808 04:47:47.545471 28611 tablet_replica.cc:330] stopping tablet replica
I20260808 04:47:47.545784 28611 raft_consensus.cc:2229] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:47:47.546080 28611 raft_consensus.cc:2258] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:47:47.547209 28611 ts_tablet_manager.cc:1905] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:47:47.550931 28611 ts_tablet_manager.cc:1918] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:47:47.551028 28611 log.cc:1196] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/f85f41c74cc847eba4825778913b3f81
I20260808 04:47:47.551326 28611 ts_tablet_manager.cc:1939] T f85f41c74cc847eba4825778913b3f81 P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Deleting consensus metadata
I20260808 04:47:47.553100 28237 catalog_manager.cc:4841] TS 6cbe985c1e4f4d898bc68b0f2c2bd8fc (127.4.217.1:33689): tablet f85f41c74cc847eba4825778913b3f81 (table test-table [id=ebb4cb39ea8a4f47b932272b5e36a44e]) successfully deleted
W20260808 04:47:47.632396 28604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:47.632649 28604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:47.632728 28604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:47.637355 28604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:47.637427 28604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:47.637519 28604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:47:47.642681 28604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=34899
--tserver_master_addrs=127.4.217.62:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:47.643630 28604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:47.644598 28604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:47.654253 28628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:47.654268 28626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:47.654209 28622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:47.654296 28604 server_base.cc:1034] running on GCE node
I20260808 04:47:47.654834 28604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:47.655493 28604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:47.656682 28604 hybrid_clock.cc:648] HybridClock initialized: now 1786164467656664 us; error 32 us; skew 500 ppm
Aug 08 04:47:47 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:47.659271 28604 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:47:47.660060 28604 webserver.cc:460] Webserver started at http://127.4.217.2:34899/ using document root <none> and password file <none>
I20260808 04:47:47.660559 28604 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:47.660629 28604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:47.664111 28604 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:47:47.666136 28639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:47.667265 28604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:47:47.667361 28604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "a0ac27da05a34d88be7e4e25077765ac"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "81c6e74631c28239bf1eae2d7e8a2c1d"
server_key_iv: "27e10afb66c5f96682115c52dd1eade9"
server_key_version: "encryptionkey@0"
I20260808 04:47:47.667820 28604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:47.691267 28604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:47.693805 28604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:47.693948 28604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:47.694446 28604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:47.695195 28604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:47.695276 28604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:47.695349 28604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:47.695394 28604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:47.704756 28604 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:37195
I20260808 04:47:47.704804 28842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:37195 every 8 connection(s)
I20260808 04:47:47.705771 28604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:47:47.706080 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 28604
I20260808 04:47:47.706607 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:44435
--local_ip_for_outbound_sockets=127.4.217.3
--tserver_master_addrs=127.4.217.62:34733
--webserver_port=41985
--webserver_interface=127.4.217.3
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:47:47 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:47.720204 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.708085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:49151 (local address 127.4.217.62:34733)
0808 04:47:47.708288 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:47.708295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:47.709213 (+ 918us) server_negotiation.cc:407] Connection header received
0808 04:47:47.710496 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:47.710499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:47.710564 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:47.710677 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:47.711827 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.712369 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.713559 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.713806 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.713922 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.716644 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:47.716647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:47.716649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:47.716675 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:47.718163 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.718679 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.718687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.718688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.718764 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.719102 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.719106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.719108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.719282 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:47.719389 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:47.719892 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 04:47:47.720025 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:47:47.721128 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.708467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:49151)
0808 04:47:47.709068 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:47.709110 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:47.710285 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:47.710856 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:47.710862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:47.711212 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:47.711673 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.711686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.712502 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.712506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.713424 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.713434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.713963 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.713967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.714136 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.714763 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:47.714779 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:47.716459 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:47.718287 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:47.718293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:47.718306 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:47.718566 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:47.718858 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:47.718861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:47.718863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:47.719001 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:47.719388 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:47.719394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:47.719659 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 04:47:47.719848 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 04:47:47.720160 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":179,"threads_started":1}
I20260808 04:47:47.722440 28843 heartbeater.cc:347] Connected to a master server at 127.4.217.62:34733
I20260808 04:47:47.722678 28843 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:47.723153 28843 heartbeater.cc:511] Master 127.4.217.62:34733 requested a full tablet report, sending...
I20260808 04:47:47.724239 28257 ts_manager.cc:194] Registered new tserver with Master: a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195)
I20260808 04:47:47.725051 28257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:49151
W20260808 04:47:47.821285 28848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:47.821498 28848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:47.821542 28848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:47.825563 28848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:47.825613 28848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:47.825666 28848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:47:47.830586 28848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:44435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=41985
--tserver_master_addrs=127.4.217.62:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:47.831547 28848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:47.832566 28848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:47.842051 28857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:47.842058 28859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:47.842159 28848 server_base.cc:1034] running on GCE node
W20260808 04:47:47.842029 28863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:47.842697 28848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:47.843300 28848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:47.844491 28848 hybrid_clock.cc:648] HybridClock initialized: now 1786164467844477 us; error 30 us; skew 500 ppm
Aug 08 04:47:47 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:47.847126 28848 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:47:47.848030 28848 webserver.cc:460] Webserver started at http://127.4.217.3:41985/ using document root <none> and password file <none>
I20260808 04:47:47.848582 28848 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:47.848690 28848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:47.852253 28848 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:47:47.854270 28874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:47.855440 28848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:47:47.855598 28848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "26d000fd1bad4f9a88294b269e2bf1e4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "2e3d7489ef02f0250a3b3906074da947"
server_key_iv: "dc6b3a2f7ca72d1c0541703e4cee6430"
server_key_version: "encryptionkey@0"
I20260808 04:47:47.856118 28848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:47.870257 28848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:47.873172 28848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:47.873322 28848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:47.873831 28848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:47.874576 28848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:47.874622 28848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:47.874660 28848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:47.874676 28848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:47.883564 28848 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:44435
I20260808 04:47:47.883589 29060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:44435 every 8 connection(s)
I20260808 04:47:47.884514 28848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
Aug 08 04:47:47 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:47:47.894116 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 28848
I20260808 04:47:47.897640 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.886274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:36923 (local address 127.4.217.62:34733)
0808 04:47:47.886424 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:47.886428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:47.887211 (+ 783us) server_negotiation.cc:407] Connection header received
0808 04:47:47.888415 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:47.888418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:47.888466 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:47.888547 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:47.889657 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.890157 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.891221 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.891424 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.891500 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.893759 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:47.893762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:47.893764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:47.893789 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:47.895596 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.896092 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.896095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.896097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.896210 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.896539 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.896550 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.896552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.896748 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:47.896886 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:47.897417 (+ 531us) server_negotiation.cc:299] Negotiation successful
0808 04:47:47.897511 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:47:47.898654 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.886632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:36923)
0808 04:47:47.887065 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:47.887105 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:47.888225 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:47.888730 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:47.888736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:47.889082 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:47.889522 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.889546 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.890262 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.890265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.891116 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.891125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.891560 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.891563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.891696 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.892178 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:47.892193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:47.893606 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:47.895687 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:47.895693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:47.895706 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:47.895965 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:47.896309 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:47.896311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:47.896313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:47.896447 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:47.896882 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:47.896887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:47.897165 (+ 278us) client_negotiation.cc:769] Sending connection context
0808 04:47:47.897374 (+ 209us) client_negotiation.cc:240] Negotiation successful
0808 04:47:47.897698 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260808 04:47:47.899928 29063 heartbeater.cc:347] Connected to a master server at 127.4.217.62:34733
I20260808 04:47:47.900164 29063 heartbeater.cc:464] Registering TS with master...
I20260808 04:47:47.900672 29063 heartbeater.cc:511] Master 127.4.217.62:34733 requested a full tablet report, sending...
I20260808 04:47:47.901831 28257 ts_manager.cc:194] Registered new tserver with Master: 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435)
I20260808 04:47:47.902549 28257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:36923
I20260808 04:47:47.903592 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.894891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.4.217.62:34733)
0808 04:47:47.895174 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:47.895178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:47.895272 (+ 94us) server_negotiation.cc:407] Connection header received
0808 04:47:47.895361 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:47.895366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:47.895418 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:47.895515 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:47.896219 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.896925 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.898119 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.898332 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.898407 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.900050 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:47.900054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:47.900059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:47.900093 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:47.901820 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.902270 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.902274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.902275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.902358 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:47.902619 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:47.902635 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:47.902639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:47.902862 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:47.903088 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:47.903259 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 04:47:47.903395 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260808 04:47:47.905431 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:47:47.909672 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.906002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.4.217.62:34733)
0808 04:47:47.906140 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:47.906144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:47.906302 (+ 158us) server_negotiation.cc:407] Connection header received
0808 04:47:47.906476 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:47.906479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:47.906580 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:47.906686 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:47:47.906921 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.907392 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.908275 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.908501 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.908633 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.908963 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:47:47.909276 (+ 313us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:47:47.909413 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:47.909483 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 04:47:47.909534 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:47:47.910275 28257 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:47:47.913498 28257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:47:47.921587 28473 tablet_service.cc:1467] Processing CreateTablet for tablet b9380ddb125249c086930278e88c54ad (DEFAULT_TABLE table=test-table [id=d9cd68af5d524728adee26e2a167c9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:47:47.922268 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9380ddb125249c086930278e88c54ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:47.925065 28397 tablet_bootstrap.cc:492] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Bootstrap starting.
I20260808 04:47:47.926149 28397 tablet_bootstrap.cc:654] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:47.927189 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.920839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.4.217.3:44435)
0808 04:47:47.921307 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:47.921319 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:47.921340 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:47:47.921409 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:47.921413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:47.921587 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:47.921765 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:47.922131 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.923186 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.924258 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.926473 (+ 2215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.926587 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.926721 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:47.926915 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 04:47:47.927042 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":173,"threads_started":1}
I20260808 04:47:47.927225 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.920367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:44435 (local address 127.0.0.1:57878)
0808 04:47:47.920614 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:47.920626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:47.920728 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:47.921867 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:47.921875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:47.921903 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:47.921996 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.922002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.923332 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.923334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.924121 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.924130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.926610 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.926614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.926726 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.926775 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 04:47:47.926869 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:47:47.927087 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
I20260808 04:47:47.927924 28397 tablet_bootstrap.cc:492] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: No bootstrap required, opened a new log
I20260808 04:47:47.928014 28397 ts_tablet_manager.cc:1397] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:47:47.928432 28397 raft_consensus.cc:348] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:47.928591 28397 raft_consensus.cc:374] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:47.928639 28397 raft_consensus.cc:729] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cbe985c1e4f4d898bc68b0f2c2bd8fc, State: Initialized, Role: FOLLOWER
I20260808 04:47:47.928828 28397 consensus_queue.cc:260] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:47.929107 28397 ts_tablet_manager.cc:1428] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:47:47.930011 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.920007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:37195 (local address 127.0.0.1:48008)
0808 04:47:47.920210 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:47.920224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:47.920304 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:47.925192 (+ 4888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:47.925198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:47.925223 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:47.925334 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.925344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.927060 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.927063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.928460 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:47.928473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.929572 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.929579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:47.929701 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.929726 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:47:47.929811 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:47:47.929890 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260808 04:47:47.930085 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:47.920220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.4.217.2:37195)
0808 04:47:47.920612 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:47.920625 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:47.920650 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:47:47.920734 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:47.920739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:47.920999 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:47.925522 (+ 4523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:47.925755 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.926902 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.928616 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:47.929454 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:47.929570 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:47.929713 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:47.929841 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 04:47:47.929931 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":131,"threads_started":1}
I20260808 04:47:47.930765 28952 tablet_service.cc:1467] Processing CreateTablet for tablet b9380ddb125249c086930278e88c54ad (DEFAULT_TABLE table=test-table [id=d9cd68af5d524728adee26e2a167c9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:47:47.931903 28952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9380ddb125249c086930278e88c54ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:47.932757 28727 tablet_service.cc:1467] Processing CreateTablet for tablet b9380ddb125249c086930278e88c54ad (DEFAULT_TABLE table=test-table [id=d9cd68af5d524728adee26e2a167c9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:47:47.933622 28727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9380ddb125249c086930278e88c54ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:47:47.939090 29085 tablet_bootstrap.cc:492] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: Bootstrap starting.
I20260808 04:47:47.939991 29086 tablet_bootstrap.cc:492] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: Bootstrap starting.
I20260808 04:47:47.941839 29085 tablet_bootstrap.cc:654] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:47.942574 29085 log.cc:824] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:47.942618 29086 tablet_bootstrap.cc:654] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: Neither blocks nor log segments found. Creating new log.
I20260808 04:47:47.943290 29086 log.cc:824] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:47.944505 29085 tablet_bootstrap.cc:492] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: No bootstrap required, opened a new log
I20260808 04:47:47.944711 29085 ts_tablet_manager.cc:1397] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:47:47.945147 29086 tablet_bootstrap.cc:492] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: No bootstrap required, opened a new log
I20260808 04:47:47.945328 29086 ts_tablet_manager.cc:1397] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260808 04:47:47.948343 29086 raft_consensus.cc:348] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:47.948342 29085 raft_consensus.cc:348] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:47.948575 29086 raft_consensus.cc:374] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:47.948611 29085 raft_consensus.cc:374] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:47:47.948637 29086 raft_consensus.cc:729] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ac27da05a34d88be7e4e25077765ac, State: Initialized, Role: FOLLOWER
I20260808 04:47:47.948696 29085 raft_consensus.cc:729] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d000fd1bad4f9a88294b269e2bf1e4, State: Initialized, Role: FOLLOWER
I20260808 04:47:47.949103 29086 consensus_queue.cc:260] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:47.949144 29085 consensus_queue.cc:260] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:47.949743 28843 heartbeater.cc:503] Master 127.4.217.62:34733 was elected leader, sending a full tablet report...
I20260808 04:47:47.949793 29063 heartbeater.cc:503] Master 127.4.217.62:34733 was elected leader, sending a full tablet report...
I20260808 04:47:47.950099 29086 ts_tablet_manager.cc:1428] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260808 04:47:47.951088 29085 ts_tablet_manager.cc:1428] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260808 04:47:47.956754 28844 tablet.cc:2369] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:47:48.001811 29091 raft_consensus.cc:482] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:47:48.002012 29091 raft_consensus.cc:504] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:48.003263 29091 leader_election.cc:288] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195), 6cbe985c1e4f4d898bc68b0f2c2bd8fc (127.4.217.1:33689)
W20260808 04:47:48.004688 28600 tablet.cc:2369] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:47:48.008119 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.003656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:52865 (local address 127.4.217.2:37195)
0808 04:47:48.003797 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:48.003804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:48.004206 (+ 402us) server_negotiation.cc:407] Connection header received
0808 04:47:48.004364 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:48.004367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:48.004420 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:48.004508 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:48.004798 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.005456 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.006840 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.007444 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.007555 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:48.007625 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:48.007911 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 04:47:48.007987 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:47:48.008119 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.003641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:49083 (local address 127.4.217.1:33689)
0808 04:47:48.003772 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:48.003776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:48.003791 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:47:48.003931 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:48.003934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:48.003997 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:48.004106 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:47:48.004408 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.005228 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.006749 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.007439 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.007564 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:48.007717 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:48.007911 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 04:47:48.007987 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:47:48.008186 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.003506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33689 (local address 127.4.217.3:49083)
0808 04:47:48.003675 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.003692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:48.003796 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:48.004121 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:48.004124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:48.004144 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:48.004287 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:48.004293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.005357 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.005363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:48.006613 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:48.006624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.007574 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.007578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:48.007716 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:48.007784 (+ 68us) client_negotiation.cc:769] Sending connection context
0808 04:47:48.007872 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 04:47:48.008010 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":58}
I20260808 04:47:48.008201 29095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.003506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:37195 (local address 127.4.217.3:52865)
0808 04:47:48.004102 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.004115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:48.004241 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:48.004521 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:48.004524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:48.004540 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:47:48.004660 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:48.004667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.005575 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.005578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:48.006723 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:48.006733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.007574 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.007578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:48.007712 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:48.007776 (+ 64us) client_negotiation.cc:769] Sending connection context
0808 04:47:48.007875 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 04:47:48.008010 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":264,"thread_start_us":165,"threads_started":1}
I20260808 04:47:48.008860 28506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9380ddb125249c086930278e88c54ad" candidate_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" is_pre_election: true
I20260808 04:47:48.008935 28763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9380ddb125249c086930278e88c54ad" candidate_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ac27da05a34d88be7e4e25077765ac" is_pre_election: true
I20260808 04:47:48.009202 28506 raft_consensus.cc:2454] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d000fd1bad4f9a88294b269e2bf1e4 in term 0.
I20260808 04:47:48.009202 28763 raft_consensus.cc:2454] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d000fd1bad4f9a88294b269e2bf1e4 in term 0.
I20260808 04:47:48.009698 28883 leader_election.cc:302] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26d000fd1bad4f9a88294b269e2bf1e4, 6cbe985c1e4f4d898bc68b0f2c2bd8fc, a0ac27da05a34d88be7e4e25077765ac; no voters:
I20260808 04:47:48.010035 29091 raft_consensus.cc:2781] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:47:48.010146 29091 raft_consensus.cc:482] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:47:48.010210 29091 raft_consensus.cc:3037] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:48.011278 29091 raft_consensus.cc:504] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:48.011641 29091 leader_election.cc:288] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [CANDIDATE]: Term 1 election: Requested vote from peers a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195), 6cbe985c1e4f4d898bc68b0f2c2bd8fc (127.4.217.1:33689)
I20260808 04:47:48.012039 28763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9380ddb125249c086930278e88c54ad" candidate_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ac27da05a34d88be7e4e25077765ac"
I20260808 04:47:48.012008 28506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9380ddb125249c086930278e88c54ad" candidate_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc"
I20260808 04:47:48.012163 28506 raft_consensus.cc:3037] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:48.012174 28763 raft_consensus.cc:3037] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:47:48.013336 28506 raft_consensus.cc:2454] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d000fd1bad4f9a88294b269e2bf1e4 in term 1.
I20260808 04:47:48.013348 28763 raft_consensus.cc:2454] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d000fd1bad4f9a88294b269e2bf1e4 in term 1.
I20260808 04:47:48.013659 28881 leader_election.cc:302] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d000fd1bad4f9a88294b269e2bf1e4, a0ac27da05a34d88be7e4e25077765ac; no voters:
I20260808 04:47:48.013847 29091 raft_consensus.cc:2781] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:47:48.014060 29091 raft_consensus.cc:686] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 LEADER]: Becoming Leader. State: Replica: 26d000fd1bad4f9a88294b269e2bf1e4, State: Running, Role: LEADER
I20260808 04:47:48.014402 29091 consensus_queue.cc:237] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:48.017493 28253 catalog_manager.cc:5495] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3). New cstate: current_term: 1 leader_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } health_report { overall_health: UNKNOWN } } }
I20260808 04:47:48.032534 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.029112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.4.217.3:44435)
0808 04:47:48.029248 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:48.029252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:48.029266 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:47:48.029418 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:48.029420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:48.029474 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:48.029558 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:47:48.029764 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.030240 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.031120 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:48.031320 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:48.031398 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:48.031772 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:47:48.032164 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:47:48.032256 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:48.032316 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 04:47:48.032364 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:47:48.038292 28506 raft_consensus.cc:1261] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Refusing update from remote peer 26d000fd1bad4f9a88294b269e2bf1e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:47:48.038345 28763 raft_consensus.cc:1261] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [term 1 FOLLOWER]: Refusing update from remote peer 26d000fd1bad4f9a88294b269e2bf1e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:47:48.039009 29091 consensus_queue.cc:1035] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:47:48.039213 29096 consensus_queue.cc:1035] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:47:48.045320 29098 mvcc.cc:205] Tried to move back new op lower bound from 7316129661077200896 to 7316129660989816832. Current Snapshot: MvccSnapshot[applied={T|T < 7316129661077200896}]
I20260808 04:47:48.054620 28253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:47:48.054798 28253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:47:48.057292 28253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Sending DeleteTablet for 3 replicas of tablet b9380ddb125249c086930278e88c54ad
I20260808 04:47:48.058023 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet b9380ddb125249c086930278e88c54ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:48 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:59632
I20260808 04:47:48.058012 28952 tablet_service.cc:1514] Processing DeleteTablet for tablet b9380ddb125249c086930278e88c54ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:48 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:57878
I20260808 04:47:48.058013 28727 tablet_service.cc:1514] Processing DeleteTablet for tablet b9380ddb125249c086930278e88c54ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:47:48 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:48008
I20260808 04:47:48.058429 29112 tablet_replica.cc:330] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: stopping tablet replica
I20260808 04:47:48.058444 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 28324
I20260808 04:47:48.058560 29112 raft_consensus.cc:2229] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:47:48.058684 29112 raft_consensus.cc:2258] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:47:48.068636 29116 tablet_replica.cc:330] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: stopping tablet replica
I20260808 04:47:48.069026 29126 tablet_replica.cc:330] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: stopping tablet replica
I20260808 04:47:48.069088 29116 raft_consensus.cc:2229] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:47:48.069425 29126 raft_consensus.cc:2229] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:47:48.069568 29116 raft_consensus.cc:2258] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:47:48.069952 29126 raft_consensus.cc:2258] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:47:48.071803 29116 ts_tablet_manager.cc:1905] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:47:48.071802 29126 ts_tablet_manager.cc:1905] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:47:48.074811 29116 ts_tablet_manager.cc:1918] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:47:48.074929 29116 log.cc:1196] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/b9380ddb125249c086930278e88c54ad
I20260808 04:47:48.075304 29116 ts_tablet_manager.cc:1939] T b9380ddb125249c086930278e88c54ad P a0ac27da05a34d88be7e4e25077765ac: Deleting consensus metadata
I20260808 04:47:48.075752 29126 ts_tablet_manager.cc:1918] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:47:48.075914 29126 log.cc:1196] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/b9380ddb125249c086930278e88c54ad
I20260808 04:47:48.076324 29126 ts_tablet_manager.cc:1939] T b9380ddb125249c086930278e88c54ad P 26d000fd1bad4f9a88294b269e2bf1e4: Deleting consensus metadata
I20260808 04:47:48.076658 28236 catalog_manager.cc:4841] TS a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195): tablet b9380ddb125249c086930278e88c54ad (table test-table [id=d9cd68af5d524728adee26e2a167c9d4]) successfully deleted
I20260808 04:47:48.077690 28234 catalog_manager.cc:4841] TS 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435): tablet b9380ddb125249c086930278e88c54ad (table test-table [id=d9cd68af5d524728adee26e2a167c9d4]) successfully deleted
W20260808 04:47:48.078238 28237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:33689 (error 108)
I20260808 04:47:48.078594 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 28604
I20260808 04:47:48.084182 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.079027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33689 (local address 127.0.0.1:59642)
0808 04:47:48.083987 (+ 4960us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.084086 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4855}
W20260808 04:47:48.084558 28237 catalog_manager.cc:4584] TS 6cbe985c1e4f4d898bc68b0f2c2bd8fc (127.4.217.1:33689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9380ddb125249c086930278e88c54ad: Network error: Client connection negotiation failed: client connection to 127.4.217.1:33689: connect: Connection refused (error 111)
I20260808 04:47:48.094976 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 28848
I20260808 04:47:48.107867 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 28208
I20260808 04:47:48.119150 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:34733
--webserver_interface=127.4.217.62
--webserver_port=42929
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:34733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:47:48.233770 29128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:48.234025 29128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:48.234074 29128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:48.237982 29128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:47:48.238041 29128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:48.238060 29128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:48.238076 29128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:47:48.238091 29128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:47:48.243275 29128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:34733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=42929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:48.244297 29128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:48.245338 29128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:48.255580 29146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:48.255621 29137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:48.255870 29140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:48.256664 29128 server_base.cc:1034] running on GCE node
I20260808 04:47:48.257427 29128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:48.258936 29128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:48.260121 29128 hybrid_clock.cc:648] HybridClock initialized: now 1786164468260092 us; error 47 us; skew 500 ppm
Aug 08 04:47:48 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:48.262838 29128 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:47:48.263672 29128 webserver.cc:460] Webserver started at http://127.4.217.62:42929/ using document root <none> and password file <none>
I20260808 04:47:48.264200 29128 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:48.264257 29128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:48.268093 29128 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:47:48.270212 29157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:48.271425 29128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:47:48.271528 29128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "426bb1182fa74334a9e89e654ad67fa4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "bae2c3ed3c6f49359c5045883174b9ed"
server_key_iv: "8cb14fed279c10ea49e64a0d9b1cee22"
server_key_version: "encryptionkey@0"
I20260808 04:47:48.272033 29128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:48.286579 29128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:48.289409 29128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:48.289572 29128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:48.297039 29128 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:34733
I20260808 04:47:48.297335 29250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:34733 every 8 connection(s)
I20260808 04:47:48.298156 29128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:47:48.302438 29251 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:47:48.304245 29251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Bootstrap starting.
I20260808 04:47:48.305922 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 29128
I20260808 04:47:48.306425 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:33689
--local_ip_for_outbound_sockets=127.4.217.1
--tserver_master_addrs=127.4.217.62:34733
--webserver_port=44217
--webserver_interface=127.4.217.1
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:47:48.322813 29251 log.cc:824] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:48.334482 29251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:47:48.334883 29251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4: Bootstrap complete.
I20260808 04:47:48.339210 29251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:48.339630 29251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426bb1182fa74334a9e89e654ad67fa4, State: Initialized, Role: FOLLOWER
I20260808 04:47:48.340241 29251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:48.340361 29251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:47:48.340446 29251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:47:48.340570 29251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:47:48.342595 29251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:48.342929 29251 leader_election.cc:302] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 426bb1182fa74334a9e89e654ad67fa4; no voters:
I20260808 04:47:48.343263 29251 leader_election.cc:288] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:47:48.343408 29255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:47:48.343678 29255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [term 3 LEADER]: Becoming Leader. State: Replica: 426bb1182fa74334a9e89e654ad67fa4, State: Running, Role: LEADER
I20260808 04:47:48.343981 29255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } }
I20260808 04:47:48.344453 29251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:47:48.344981 29255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 426bb1182fa74334a9e89e654ad67fa4. Latest consensus state: current_term: 3 leader_uuid: "426bb1182fa74334a9e89e654ad67fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } } }
I20260808 04:47:48.345104 29255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:48.345899 29261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:47:48.346307 29256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "426bb1182fa74334a9e89e654ad67fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426bb1182fa74334a9e89e654ad67fa4" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34733 } } }
I20260808 04:47:48.346402 29256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426bb1182fa74334a9e89e654ad67fa4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:47:48.351078 29261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:47:48.351835 29261 catalog_manager.cc:1259] Loaded cluster ID: 444995ef7fa3404fb5ae45ead5fa480e
I20260808 04:47:48.351909 29261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 04:47:48.352758 29261 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f21f2a829ad google::LogMessage::Fail() at ??:0
@ 0x7f21f2a8588a google::LogMessage::SendToLog() at ??:0
@ 0x7f21f2a824de google::LogMessage::Flush() at ??:0
@ 0x7f21f2a83a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f21fd3c1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f21fd3c2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f21fd3bf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f21fd403640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21f4cd43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f21f4cd4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f21f4cd63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21f4cc5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f21f63096db start_thread at ??:0
@ 0x7f21f158061f clone at ??:0
W20260808 04:47:48.439667 29253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:48.439935 29253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:48.440050 29253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:48.444085 29253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:48.444162 29253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:48.444268 29253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:47:48.449496 29253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:33689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=44217
--tserver_master_addrs=127.4.217.62:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:48.450529 29253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:48.451548 29253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:48.462225 29328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:48.462381 29253 server_base.cc:1034] running on GCE node
W20260808 04:47:48.462232 29322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:48.462244 29325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:48.463151 29253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:48.464030 29253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:48.465234 29253 hybrid_clock.cc:648] HybridClock initialized: now 1786164468465201 us; error 44 us; skew 500 ppm
Aug 08 04:47:48 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:48.468528 29253 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:47:48.469493 29253 webserver.cc:460] Webserver started at http://127.4.217.1:44217/ using document root <none> and password file <none>
I20260808 04:47:48.470084 29253 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:48.470149 29253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:48.474267 29253 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:47:48.476575 29342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:48.478094 29253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:47:48.478216 29253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "6f8616ea813494868cb7a19ea7888ee7"
server_key_iv: "2e0f1cf26eba55d39cb46ba205ff632d"
server_key_version: "encryptionkey@0"
I20260808 04:47:48.478904 29253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:48.508165 29253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:48.512028 29253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:48.512238 29253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:48.513025 29253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:48.514410 29352 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:47:48.518671 29253 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:47:48.518747 29253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260808 04:47:48.518813 29253 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:47:48.521138 29253 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:47:48.521200 29253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260808 04:47:48.521497 29352 tablet_bootstrap.cc:492] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Bootstrap starting.
I20260808 04:47:48.532593 29253 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:33689
I20260808 04:47:48.532773 29528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:33689 every 8 connection(s)
I20260808 04:47:48.533728 29253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:47:48.534783 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 29253
I20260808 04:47:48.535254 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:37195
--local_ip_for_outbound_sockets=127.4.217.2
--tserver_master_addrs=127.4.217.62:34733
--webserver_port=34899
--webserver_interface=127.4.217.2
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:47:48.543545 29352 log.cc:824] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:47:48.553202 29352 tablet_bootstrap.cc:492] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:47:48.553601 29352 tablet_bootstrap.cc:492] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Bootstrap complete.
I20260808 04:47:48.555197 29352 ts_tablet_manager.cc:1397] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.011s
I20260808 04:47:48.558471 29352 raft_consensus.cc:348] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:48.559017 29352 raft_consensus.cc:729] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cbe985c1e4f4d898bc68b0f2c2bd8fc, State: Initialized, Role: FOLLOWER
I20260808 04:47:48.559548 29352 consensus_queue.cc:260] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:48.560590 29352 ts_tablet_manager.cc:1428] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc: Time spent starting tablet: real 0.005s user 0.004s sys 0.003s
W20260808 04:47:48.595504 29536 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:47:48.538744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:44019)
0808 04:47:48.539548 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.539609 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:48.540655 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:48.594656 (+ 54001us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":190,"threads_started":1}
W20260808 04:47:48.596027 29530 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:34733 (0 consecutive failures): Network error: Failed to ping master at 127.4.217.62:34733: Client connection negotiation failed: client connection to 127.4.217.62:34733: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:47:48.596884 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.596570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:43977)
0808 04:47:48.596702 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.596765 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:47:48.597553 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.597333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:58917)
0808 04:47:48.597451 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.597501 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:47:48.597728 29530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:47:48.661247 29534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:48.661509 29534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:48.661582 29534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:48.665570 29534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:48.665644 29534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:48.665726 29534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:47:48.670815 29534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=34899
--tserver_master_addrs=127.4.217.62:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:48.671803 29534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:48.672856 29534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:47:48.682596 29534 server_base.cc:1034] running on GCE node
W20260808 04:47:48.682618 29554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:48.682618 29557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:48.682890 29555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:48.683328 29534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:48.684010 29534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:48.685194 29534 hybrid_clock.cc:648] HybridClock initialized: now 1786164468685165 us; error 46 us; skew 500 ppm
Aug 08 04:47:48 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:48.687909 29534 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:47:48.688759 29534 webserver.cc:460] Webserver started at http://127.4.217.2:34899/ using document root <none> and password file <none>
I20260808 04:47:48.689368 29534 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:48.689462 29534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:48.693082 29534 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 04:47:48.695017 29568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:48.696197 29534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:47:48.696332 29534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "a0ac27da05a34d88be7e4e25077765ac"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "81c6e74631c28239bf1eae2d7e8a2c1d"
server_key_iv: "27e10afb66c5f96682115c52dd1eade9"
server_key_version: "encryptionkey@0"
I20260808 04:47:48.696918 29534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:48.710788 29534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:48.713421 29534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:48.713632 29534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:48.714190 29534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:48.714987 29534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:48.715062 29534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:48.715128 29534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:48.715180 29534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:48.724059 29534 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:37195
I20260808 04:47:48.724116 29758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:37195 every 8 connection(s)
I20260808 04:47:48.725010 29534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:47:48.726959 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 29534
I20260808 04:47:48.727468 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:44435
--local_ip_for_outbound_sockets=127.4.217.3
--tserver_master_addrs=127.4.217.62:34733
--webserver_port=41985
--webserver_interface=127.4.217.3
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:47:48.728966 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.727076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:35263)
0808 04:47:48.727665 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.728219 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":150,"threads_started":1}
W20260808 04:47:48.729488 29759 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:34733 (0 consecutive failures): Network error: Failed to ping master at 127.4.217.62:34733: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
I20260808 04:47:48.730110 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.729871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:53201)
0808 04:47:48.729976 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.730026 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 04:47:48.730821 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.730638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:41435)
0808 04:47:48.730735 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.730775 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 04:47:48.731073 29759 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:47:48.842971 29762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:47:48.843220 29762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:47:48.843261 29762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:47:48.847249 29762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:47:48.847304 29762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:47:48.847409 29762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:47:48.852986 29762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:44435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=41985
--tserver_master_addrs=127.4.217.62:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:47:48.853972 29762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:47:48.854977 29762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:47:48.864651 29770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:48.864658 29769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:47:48.864701 29774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:47:48.864873 29762 server_base.cc:1034] running on GCE node
I20260808 04:47:48.865343 29762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:47:48.865901 29762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:47:48.867107 29762 hybrid_clock.cc:648] HybridClock initialized: now 1786164468867093 us; error 26 us; skew 500 ppm
Aug 08 04:47:48 dist-test-slave-2kcd krb5kdc[27269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:47:48.869922 29762 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:47:48.870826 29762 webserver.cc:460] Webserver started at http://127.4.217.3:41985/ using document root <none> and password file <none>
I20260808 04:47:48.871384 29762 fs_manager.cc:359] Metadata directory not provided
I20260808 04:47:48.871480 29762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:47:48.875177 29762 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:47:48.877208 29790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:48.878397 29762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:47:48.878571 29762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "26d000fd1bad4f9a88294b269e2bf1e4"
format_stamp: "Formatted at 2026-08-08 04:47:46 on dist-test-slave-2kcd"
server_key: "2e3d7489ef02f0250a3b3906074da947"
server_key_iv: "dc6b3a2f7ca72d1c0541703e4cee6430"
server_key_version: "encryptionkey@0"
I20260808 04:47:48.879158 29762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:47:48.894733 29762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:47:48.897466 29762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:47:48.897675 29762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:47:48.898245 29762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:47:48.899029 29762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:47:48.899108 29762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:48.899180 29762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:47:48.899225 29762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:47:48.908373 29762 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:44435
I20260808 04:47:48.908399 29982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:44435 every 8 connection(s)
I20260808 04:47:48.909350 29762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:47:48.912927 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.911326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:45047)
0808 04:47:48.911763 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.912263 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
W20260808 04:47:48.913373 29987 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:34733 (0 consecutive failures): Network error: Failed to ping master at 127.4.217.62:34733: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
I20260808 04:47:48.913947 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.913750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:45067)
0808 04:47:48.913848 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.913895 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 04:47:48.914556 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:48.914377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:45659)
0808 04:47:48.914471 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:48.914511 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:47:48.914779 29987 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:47:48.916608 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 29762
I20260808 04:47:49.598857 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:49.598331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:55537)
0808 04:47:49.598680 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:49.598740 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":76,"threads_started":1}
I20260808 04:47:49.731922 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:49.731550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:46595)
0808 04:47:49.731775 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:49.731827 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260808 04:47:49.826778 30003 raft_consensus.cc:482] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:47:49.827070 30003 raft_consensus.cc:504] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:49.828155 30003 leader_election.cc:288] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195), 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435)
Aug 08 04:47:49 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.2@KRBTEST.COM
Aug 08 04:47:49 dist-test-slave-2kcd krb5kdc[27269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.3@KRBTEST.COM
I20260808 04:47:49.841847 30005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:49.828597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:33121 (local address 127.4.217.2:37195)
0808 04:47:49.829003 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:49.829023 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:49.829310 (+ 287us) server_negotiation.cc:407] Connection header received
0808 04:47:49.830004 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:49.830009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:49.830320 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:49.830612 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:49.831656 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:49.832614 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:49.833856 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:49.834327 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:49.834508 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:49.837415 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:49.837422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:49.837439 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:49.837479 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:49.839286 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:49.839829 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:49.839833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:49.839837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:49.839946 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:49.840292 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:49.840295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:49.840297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:49.840687 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:49.840851 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:49.841385 (+ 534us) server_negotiation.cc:299] Negotiation successful
0808 04:47:49.841646 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":90,"threads_started":1}
I20260808 04:47:49.841862 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:49.828398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:37195 (local address 127.4.217.1:33121)
0808 04:47:49.829180 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:49.829192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:49.829321 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:49.830597 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:49.830601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:49.831003 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:49.831467 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:49.831481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:49.832775 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:49.832802 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:49.833731 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:49.833741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:49.834463 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:49.834467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:49.834654 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:49.835402 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:49.835414 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:49.837060 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:49.839421 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:49.839428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:49.839441 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:49.839697 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:49.840047 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:49.840052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:49.840054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:49.840190 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:49.840829 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:49.840836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:49.841065 (+ 229us) client_negotiation.cc:769] Sending connection context
0808 04:47:49.841247 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 04:47:49.841604 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":610,"thread_start_us":61,"threads_started":1}
I20260808 04:47:49.841912 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:49.828583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:52999 (local address 127.4.217.3:44435)
0808 04:47:49.828939 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0808 04:47:49.828956 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:47:49.829007 (+ 51us) server_negotiation.cc:407] Connection header received
0808 04:47:49.829713 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:47:49.829719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:47:49.830090 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:47:49.830335 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:47:49.831657 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:49.832615 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:49.833900 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:47:49.834337 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:47:49.834521 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:49.837749 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:47:49.837754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:47:49.837764 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:47:49.837792 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:47:49.839637 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:49.840097 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:49.840103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:49.840108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:49.840226 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:47:49.840525 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:47:49.840528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:47:49.840530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:47:49.840962 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:47:49.841101 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:47:49.841443 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 04:47:49.841731 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":129,"threads_started":1}
I20260808 04:47:49.842509 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:49.828398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:44435 (local address 127.4.217.1:52999)
0808 04:47:49.828595 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:49.828611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:47:49.828734 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:47:49.830447 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:47:49.830456 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:47:49.831052 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:47:49.831466 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:49.831481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:49.832764 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:49.832769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:49.833738 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:47:49.833748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:47:49.834465 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:47:49.834468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:47:49.834656 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:47:49.835357 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:47:49.835373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:47:49.837535 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:47:49.839761 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:49.839768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:49.839770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:49.839970 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:49.840328 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:47:49.840332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:47:49.840334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:47:49.840424 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:47:49.841076 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:47:49.841079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:47:49.841174 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 04:47:49.841289 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 04:47:49.841704 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":288,"spinlock_wait_cycles":23168}
W20260808 04:47:49.844408 29345 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
W20260808 04:47:49.844564 29343 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
I20260808 04:47:49.844619 29345 leader_election.cc:302] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cbe985c1e4f4d898bc68b0f2c2bd8fc; no voters: 26d000fd1bad4f9a88294b269e2bf1e4, a0ac27da05a34d88be7e4e25077765ac
I20260808 04:47:49.844908 30003 raft_consensus.cc:2726] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:47:49.915742 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:49.915252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:42411)
0808 04:47:49.915592 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:49.915645 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":78,"threads_started":1}
I20260808 04:47:50.600330 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:50.599826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:49065)
0808 04:47:50.600169 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:50.600228 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":80,"threads_started":1}
I20260808 04:47:50.733096 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:50.732667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:60727)
0808 04:47:50.732946 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:50.732998 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":72,"threads_started":1}
I20260808 04:47:50.916963 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:50.916506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:35251)
0808 04:47:50.916748 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:50.916859 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260808 04:47:51.601606 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:51.601153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:40295)
0808 04:47:51.601455 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:51.601506 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":129,"threads_started":1}
I20260808 04:47:51.644496 30018 raft_consensus.cc:482] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:47:51.644663 30018 raft_consensus.cc:504] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:51.645157 30018 leader_election.cc:288] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195), 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435)
W20260808 04:47:51.646011 29345 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
W20260808 04:47:51.646162 29343 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
I20260808 04:47:51.646251 29343 leader_election.cc:302] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cbe985c1e4f4d898bc68b0f2c2bd8fc; no voters: 26d000fd1bad4f9a88294b269e2bf1e4, a0ac27da05a34d88be7e4e25077765ac
I20260808 04:47:51.646445 30018 raft_consensus.cc:2726] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:47:51.734146 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:51.733748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:46821)
0808 04:47:51.733992 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:51.734042 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260808 04:47:51.918108 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:51.917678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:51231)
0808 04:47:51.917956 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:51.918011 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260808 04:47:52.602962 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:52.602383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:34053)
0808 04:47:52.602702 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:52.602779 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":80,"threads_started":1}
I20260808 04:47:52.735301 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:52.734890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:50955)
0808 04:47:52.735155 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:52.735207 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260808 04:47:52.919219 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:52.918836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:51827)
0808 04:47:52.919074 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:52.919126 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260808 04:47:53.337426 30029 raft_consensus.cc:482] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:47:53.337561 30029 raft_consensus.cc:504] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:53.338052 30029 leader_election.cc:288] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195), 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435)
W20260808 04:47:53.338864 29345 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
W20260808 04:47:53.339011 29343 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
I20260808 04:47:53.339088 29343 leader_election.cc:302] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cbe985c1e4f4d898bc68b0f2c2bd8fc; no voters: 26d000fd1bad4f9a88294b269e2bf1e4, a0ac27da05a34d88be7e4e25077765ac
I20260808 04:47:53.339222 30029 raft_consensus.cc:2726] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:47:53.604378 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:53.603869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:44363)
0808 04:47:53.604215 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:53.604281 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
I20260808 04:47:53.736433 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:53.736031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:58225)
0808 04:47:53.736283 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:53.736337 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260808 04:47:53.920387 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:53.919980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:35761)
0808 04:47:53.920238 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:53.920294 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260808 04:47:54.605597 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:54.605199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:59843)
0808 04:47:54.605439 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:54.605493 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260808 04:47:54.737749 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:54.737216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:43335)
0808 04:47:54.737544 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:54.737629 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":121,"threads_started":1}
I20260808 04:47:54.921520 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:54.921126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:41619)
0808 04:47:54.921383 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:54.921436 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260808 04:47:55.606796 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:55.606379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:55839)
0808 04:47:55.606635 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:55.606689 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260808 04:47:55.739066 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:55.738560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:37389)
0808 04:47:55.738907 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:55.738968 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":79,"threads_started":1}
I20260808 04:47:55.922710 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:55.922256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:46969)
0808 04:47:55.922561 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:55.922615 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":72,"threads_started":1}
I20260808 04:47:56.217105 30047 raft_consensus.cc:482] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:47:56.217252 30047 raft_consensus.cc:504] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:56.217760 30047 leader_election.cc:288] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195), 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435)
W20260808 04:47:56.218591 29345 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
W20260808 04:47:56.218703 29343 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
I20260808 04:47:56.218920 29343 leader_election.cc:302] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cbe985c1e4f4d898bc68b0f2c2bd8fc; no voters: 26d000fd1bad4f9a88294b269e2bf1e4, a0ac27da05a34d88be7e4e25077765ac
I20260808 04:47:56.219100 30047 raft_consensus.cc:2726] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:47:56.608021 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:56.607605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:38603)
0808 04:47:56.607865 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:56.607919 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260808 04:47:56.740381 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:56.739951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:53263)
0808 04:47:56.740210 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:56.740280 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260808 04:47:56.924002 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:56.923560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:55981)
0808 04:47:56.923841 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:56.923902 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260808 04:47:57.609373 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:57.608923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:46591)
0808 04:47:57.609217 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:57.609272 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":112,"threads_started":1}
I20260808 04:47:57.741737 30056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:57.741285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:48575)
0808 04:47:57.741574 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:57.741638 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260808 04:47:57.925262 30057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:57.924839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:39895)
0808 04:47:57.925108 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:57.925157 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260808 04:47:58.610879 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:58.610386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:53943)
0808 04:47:58.610710 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:58.610775 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260808 04:47:58.743093 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:58.742654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:38825)
0808 04:47:58.742930 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:58.742993 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260808 04:47:58.927091 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:58.926390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:33909)
0808 04:47:58.926814 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:58.926943 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":143,"threads_started":1}
I20260808 04:47:59.040155 30064 raft_consensus.cc:482] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:47:59.040392 30064 raft_consensus.cc:504] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:47:59.041359 30064 leader_election.cc:288] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195), 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435)
W20260808 04:47:59.042596 29345 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
W20260808 04:47:59.042814 29343 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
I20260808 04:47:59.042985 29343 leader_election.cc:302] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cbe985c1e4f4d898bc68b0f2c2bd8fc; no voters: 26d000fd1bad4f9a88294b269e2bf1e4, a0ac27da05a34d88be7e4e25077765ac
I20260808 04:47:59.043287 30064 raft_consensus.cc:2726] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:47:59.612535 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:59.612023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:47879)
0808 04:47:59.612361 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:59.612428 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260808 04:47:59.744498 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:59.744035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:35675)
0808 04:47:59.744323 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:59.744395 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260808 04:47:59.928496 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:47:59.928062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:39609)
0808 04:47:59.928340 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 04:47:59.928404 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260808 04:48:00.613996 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:00.613501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:33427)
0808 04:48:00.613796 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:00.613859 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":115,"threads_started":1}
I20260808 04:48:00.745738 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:00.745349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:48505)
0808 04:48:00.745593 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:00.745645 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260808 04:48:00.929680 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:00.929237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:35949)
0808 04:48:00.929515 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:00.929569 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260808 04:48:01.615232 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:01.614789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:44499)
0808 04:48:01.615077 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:01.615134 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260808 04:48:01.746978 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:01.746542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:58701)
0808 04:48:01.746829 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:01.746882 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260808 04:48:01.930928 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:01.930499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:51603)
0808 04:48:01.930765 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:01.930821 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260808 04:48:02.552809 30081 raft_consensus.cc:482] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:48:02.552994 30081 raft_consensus.cc:504] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ac27da05a34d88be7e4e25077765ac" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 37195 } } peers { permanent_uuid: "26d000fd1bad4f9a88294b269e2bf1e4" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 44435 } } peers { permanent_uuid: "6cbe985c1e4f4d898bc68b0f2c2bd8fc" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33689 } }
I20260808 04:48:02.553503 30081 leader_election.cc:288] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195), 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435)
W20260808 04:48:02.554389 29343 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26d000fd1bad4f9a88294b269e2bf1e4 (127.4.217.3:44435): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
W20260808 04:48:02.554494 29345 leader_election.cc:341] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0ac27da05a34d88be7e4e25077765ac (127.4.217.2:37195): Not found: Tablet not found: b9380ddb125249c086930278e88c54ad
I20260808 04:48:02.554622 29345 leader_election.cc:302] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cbe985c1e4f4d898bc68b0f2c2bd8fc; no voters: 26d000fd1bad4f9a88294b269e2bf1e4, a0ac27da05a34d88be7e4e25077765ac
I20260808 04:48:02.554793 30081 raft_consensus.cc:2726] T b9380ddb125249c086930278e88c54ad P 6cbe985c1e4f4d898bc68b0f2c2bd8fc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:48:02.616468 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:02.616028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:53393)
0808 04:48:02.616300 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:02.616358 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260808 04:48:02.748119 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:02.747697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:50839)
0808 04:48:02.747971 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:02.748026 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260808 04:48:02.932251 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:02.931791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:41151)
0808 04:48:02.932088 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:02.932147 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
I20260808 04:48:03.618062 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:03.617600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.1:46709)
0808 04:48:03.617900 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:03.617959 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260808 04:48:03.749406 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:03.748997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.2:55121)
0808 04:48:03.749258 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:03.749310 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260808 04:48:03.927989 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 29253
I20260808 04:48:03.935488 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:03.934953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34733 (local address 127.4.217.3:37859)
0808 04:48:03.935304 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:03.935379 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":90,"threads_started":1}
I20260808 04:48:03.944600 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 29534
I20260808 04:48:03.965281 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 29762
2026-08-08T04:48:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19399 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:48:04 dist-test-slave-2kcd krb5kdc[30099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:48:04 dist-test-slave-2kcd krb5kdc[30099](info): set up 2 sockets
Aug 08 04:48:04 dist-test-slave-2kcd krb5kdc[30099](info): commencing operation
krb5kdc: starting...
W20260808 04:48:05.230816 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:48:05 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:48:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:48:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:48:05.360226 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35027
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:48:05.475989 30117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:05.476357 30117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:05.476444 30117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:05.480419 30117 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:48:05.480500 30117 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:48:05.480527 30117 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:48:05.480549 30117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:48:05.480590 30117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:05.480628 30117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:05.480660 30117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:48:05.480695 30117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:48:05.485930 30117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:05.486981 30117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:05.488021 30117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:05.496743 30136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:05.496696 30128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:05.496766 30117 server_base.cc:1034] running on GCE node
W20260808 04:48:05.496927 30130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:05.497538 30117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:05.498596 30117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:05.499791 30117 hybrid_clock.cc:648] HybridClock initialized: now 1786164485499772 us; error 30 us; skew 500 ppm
Aug 08 04:48:05 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:05.502574 30117 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:48:05.503422 30117 webserver.cc:460] Webserver started at http://127.4.217.62:33119/ using document root <none> and password file <none>
I20260808 04:48:05.503968 30117 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:05.504062 30117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:05.504295 30117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:48:05.506325 30117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "5fd3c6afdeb6491b8192fe3faea7acfd"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "b50c9600b80820c785102dcb3bc363d3"
server_key_iv: "9a9e802623378df4f5d1ab383afc95b3"
server_key_version: "encryptionkey@0"
I20260808 04:48:05.506919 30117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "5fd3c6afdeb6491b8192fe3faea7acfd"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "b50c9600b80820c785102dcb3bc363d3"
server_key_iv: "9a9e802623378df4f5d1ab383afc95b3"
server_key_version: "encryptionkey@0"
I20260808 04:48:05.510665 30117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:48:05.513219 30149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:05.514575 30117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:48:05.514719 30117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "5fd3c6afdeb6491b8192fe3faea7acfd"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "b50c9600b80820c785102dcb3bc363d3"
server_key_iv: "9a9e802623378df4f5d1ab383afc95b3"
server_key_version: "encryptionkey@0"
I20260808 04:48:05.514843 30117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:05.527866 30117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:05.530736 30117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:05.530962 30117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:05.538463 30117 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:35027
I20260808 04:48:05.538753 30237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:35027 every 8 connection(s)
I20260808 04:48:05.539516 30117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:48:05.542300 30238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:48:05.547421 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 30117
I20260808 04:48:05.547583 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:48:05.547631 30238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Bootstrap starting.
I20260808 04:48:05.547943 4964 external_mini_cluster.cc:1428] Setting key 9f26bc2a92220aedaf3a07e111e949f9
I20260808 04:48:05.550567 30238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Neither blocks nor log segments found. Creating new log.
I20260808 04:48:05.551451 30238 log.cc:824] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:48:05.553678 30238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: No bootstrap required, opened a new log
I20260808 04:48:05.557699 30238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:05.557945 30238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:48:05.558034 30238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd3c6afdeb6491b8192fe3faea7acfd, State: Initialized, Role: FOLLOWER
Aug 08 04:48:05 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:48:05.558506 30238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:05.558658 30238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:48:05.558727 30238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:48:05.558822 30238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:48:05.559949 30238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:05.560302 30238 leader_election.cc:302] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fd3c6afdeb6491b8192fe3faea7acfd; no voters:
I20260808 04:48:05.560659 30238 leader_election.cc:288] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:48:05.560708 30245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:48:05.560930 30245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 1 LEADER]: Becoming Leader. State: Replica: 5fd3c6afdeb6491b8192fe3faea7acfd, State: Running, Role: LEADER
I20260808 04:48:05.561273 30245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:05.561908 30238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:48:05.562954 30247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd3c6afdeb6491b8192fe3faea7acfd. Latest consensus state: current_term: 1 leader_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } } }
I20260808 04:48:05.563081 30247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: This master's current role is: LEADER
I20260808 04:48:05.563485 30250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } } }
I20260808 04:48:05.564357 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:05.549367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.4.217.62:35027)
0808 04:48:05.549805 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:05.549824 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:05.549863 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:48:05.550848 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:05.550876 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:05.551262 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:05.551634 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:05.552797 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:05.554266 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:05.555156 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:05.555849 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:05.556014 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:05.558853 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:05.558860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:05.558883 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:05.558909 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:05.560720 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:05.561321 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:05.561338 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:05.561343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:05.561479 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:05.561791 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:05.561796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:05.561798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:05.562241 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:05.562476 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:05.563028 (+ 552us) server_negotiation.cc:299] Negotiation successful
0808 04:48:05.563341 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
I20260808 04:48:05.563630 30250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: This master's current role is: LEADER
W20260808 04:48:05.567154 30273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:48:05.567243 30273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:48:05.567319 30260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:48:05.568068 30260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:48:05.573968 30260 catalog_manager.cc:1347] Generated new cluster ID: 11ad376ca1684b43850e7c657afcd709
I20260808 04:48:05.574048 30260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:48:05.595021 30260 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:48:05.595918 30260 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:48:05.603103 30260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Generated new TSK 0
I20260808 04:48:05.603761 30260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:48:05.639143 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35027
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:48:05.759573 30281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:05.759806 30281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:05.759856 30281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:05.764132 30281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:05.764204 30281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:05.764310 30281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:48:05.769526 30281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:05.770539 30281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:05.771543 30281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:05.781461 30291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:05.781461 30295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:05.781630 30281 server_base.cc:1034] running on GCE node
W20260808 04:48:05.781822 30290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:05.782222 30281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:05.782920 30281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:05.784132 30281 hybrid_clock.cc:648] HybridClock initialized: now 1786164485784103 us; error 46 us; skew 500 ppm
Aug 08 04:48:05 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:05.787034 30281 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:48:05.787926 30281 webserver.cc:460] Webserver started at http://127.4.217.1:41057/ using document root <none> and password file <none>
I20260808 04:48:05.788520 30281 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:05.788596 30281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:05.788879 30281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:48:05.790998 30281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "6b0aabe8e1e94d8ab1062bd0163ca555"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "dcd91eb509eac3d658d82aafd2de0fde"
server_key_iv: "65738779e57b75f7e43cfbcbd51340a6"
server_key_version: "encryptionkey@0"
I20260808 04:48:05.791589 30281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "6b0aabe8e1e94d8ab1062bd0163ca555"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "dcd91eb509eac3d658d82aafd2de0fde"
server_key_iv: "65738779e57b75f7e43cfbcbd51340a6"
server_key_version: "encryptionkey@0"
I20260808 04:48:05.795491 30281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 04:48:05.798002 30303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:05.799249 30281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:48:05.799376 30281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "6b0aabe8e1e94d8ab1062bd0163ca555"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "dcd91eb509eac3d658d82aafd2de0fde"
server_key_iv: "65738779e57b75f7e43cfbcbd51340a6"
server_key_version: "encryptionkey@0"
I20260808 04:48:05.799499 30281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:05.809568 30281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:05.812295 30281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:05.812515 30281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:05.813195 30281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:48:05.814026 30281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:48:05.814111 30281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:05.814204 30281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:48:05.814260 30281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:05.823694 30281 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:39389
I20260808 04:48:05.823722 30493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:39389 every 8 connection(s)
I20260808 04:48:05.824582 30281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:48:05.825675 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 30281
I20260808 04:48:05.825803 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:48:05.826164 4964 external_mini_cluster.cc:1428] Setting key f6f3349f23c0e9fc72f20085f8f425f4
Aug 08 04:48:05 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
I20260808 04:48:05.840482 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:05.826638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:40643 (local address 127.4.217.62:35027)
0808 04:48:05.826777 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:05.826781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:05.827778 (+ 997us) server_negotiation.cc:407] Connection header received
0808 04:48:05.829231 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:05.829235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:05.829290 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:05.829377 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:05.831001 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:05.831526 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:05.833396 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:05.833628 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:05.833730 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:05.836615 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:05.836620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:05.836625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:05.836657 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:05.838427 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:05.838944 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:05.838949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:05.838952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:05.839038 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:05.839389 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:05.839392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:05.839394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:05.839571 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:05.839685 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:05.840165 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 04:48:05.840289 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:48:05.841324 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:05.827078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:40643)
0808 04:48:05.827623 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:05.827683 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:05.829021 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:05.829563 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:05.829570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:05.830119 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:48:05.830825 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:05.830848 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:05.831658 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:05.831665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:05.833260 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:05.833277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:05.833810 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:05.833818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:05.834069 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:05.834713 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:48:05.834740 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:48:05.836444 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:48:05.838533 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:05.838538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:05.838549 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:05.838794 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:05.839137 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:05.839141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:05.839143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:05.839281 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:05.839683 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:48:05.839689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:48:05.839931 (+ 242us) client_negotiation.cc:769] Sending connection context
0808 04:48:05.840121 (+ 190us) client_negotiation.cc:240] Negotiation successful
0808 04:48:05.840434 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
I20260808 04:48:05.842597 30494 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35027
I20260808 04:48:05.842831 30494 heartbeater.cc:464] Registering TS with master...
I20260808 04:48:05.843353 30494 heartbeater.cc:511] Master 127.4.217.62:35027 requested a full tablet report, sending...
I20260808 04:48:05.844919 30184 ts_manager.cc:194] Registered new tserver with Master: 6b0aabe8e1e94d8ab1062bd0163ca555 (127.4.217.1:39389)
I20260808 04:48:05.846035 30184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:40643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:48:05.856042 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35027
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:48:05.968076 30507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:05.968308 30507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:05.968351 30507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:05.972357 30507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:05.972412 30507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:05.972467 30507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:48:05.977727 30507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:05.978699 30507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:05.979665 30507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:05.989657 30515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:05.989616 30507 server_base.cc:1034] running on GCE node
W20260808 04:48:05.989759 30521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:05.989591 30517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:05.990307 30507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:05.990942 30507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:05.992137 30507 hybrid_clock.cc:648] HybridClock initialized: now 1786164485992128 us; error 31 us; skew 500 ppm
Aug 08 04:48:05 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:05.995091 30507 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:48:05.996001 30507 webserver.cc:460] Webserver started at http://127.4.217.2:40477/ using document root <none> and password file <none>
I20260808 04:48:05.996585 30507 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:05.996678 30507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:05.996977 30507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:48:05.999033 30507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "8ed55f3ed3764245a63367fb6b6651c1"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "4d2833defded0a6f99226e2c70f46703"
server_key_iv: "4393d9e270ee4183ccdb69075b848833"
server_key_version: "encryptionkey@0"
I20260808 04:48:05.999599 30507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "8ed55f3ed3764245a63367fb6b6651c1"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "4d2833defded0a6f99226e2c70f46703"
server_key_iv: "4393d9e270ee4183ccdb69075b848833"
server_key_version: "encryptionkey@0"
I20260808 04:48:06.003525 30507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:48:06.005932 30531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:06.007130 30507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:48:06.007282 30507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "8ed55f3ed3764245a63367fb6b6651c1"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "4d2833defded0a6f99226e2c70f46703"
server_key_iv: "4393d9e270ee4183ccdb69075b848833"
server_key_version: "encryptionkey@0"
I20260808 04:48:06.007395 30507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:06.034162 30507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:06.037415 30507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:06.037652 30507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:06.038347 30507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:48:06.039284 30507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:48:06.039358 30507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:06.039428 30507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:48:06.039484 30507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:06.049322 30507 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:34085
I20260808 04:48:06.049348 30723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:34085 every 8 connection(s)
I20260808 04:48:06.050280 30507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:48:06.052810 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 30507
I20260808 04:48:06.052949 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:48:06.053309 4964 external_mini_cluster.cc:1428] Setting key 670219f4d7c72045b30844065ade4d29
Aug 08 04:48:06 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:48:06.065769 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.052221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:39835 (local address 127.4.217.62:35027)
0808 04:48:06.052411 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:06.052417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:06.053463 (+ 1046us) server_negotiation.cc:407] Connection header received
0808 04:48:06.055189 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:06.055192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:06.055248 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:06.055335 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:06.056550 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.057304 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.058478 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.058863 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.059002 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.061670 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:06.061673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:06.061675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:06.061700 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:06.063476 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:06.063980 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:06.063985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:06.063987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:06.064073 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:06.064514 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:06.064520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:06.064523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:06.064804 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:06.064925 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:06.065537 (+ 612us) server_negotiation.cc:299] Negotiation successful
0808 04:48:06.065644 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:48:06.066592 30730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.052724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:39835)
0808 04:48:06.053283 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:06.053325 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:06.054953 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:06.055534 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:06.055541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:06.055911 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:48:06.056402 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.056416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.057450 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.057453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.058351 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.058361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.059053 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.059057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.059212 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.059843 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:48:06.059860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:48:06.061492 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:48:06.063599 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:06.063605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:06.063620 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:06.063872 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:06.064172 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:06.064176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:06.064178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:06.064382 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:06.064989 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:48:06.064995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:48:06.065295 (+ 300us) client_negotiation.cc:769] Sending connection context
0808 04:48:06.065466 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 04:48:06.065716 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":164,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
I20260808 04:48:06.067798 30727 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35027
I20260808 04:48:06.068095 30727 heartbeater.cc:464] Registering TS with master...
I20260808 04:48:06.069324 30727 heartbeater.cc:511] Master 127.4.217.62:35027 requested a full tablet report, sending...
I20260808 04:48:06.070503 30184 ts_manager.cc:194] Registered new tserver with Master: 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085)
I20260808 04:48:06.071266 30184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:39835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:48:06.083606 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35027
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:48:06.195571 30734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:06.195809 30734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:06.195883 30734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:06.199869 30734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:06.199924 30734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:06.200028 30734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:48:06.205360 30734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:06.206343 30734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:06.207335 30734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:06.216993 30744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:06.217080 30741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:06.217003 30739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:06.217207 30734 server_base.cc:1034] running on GCE node
I20260808 04:48:06.217692 30734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:06.218287 30734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:06.219478 30734 hybrid_clock.cc:648] HybridClock initialized: now 1786164486219459 us; error 33 us; skew 500 ppm
Aug 08 04:48:06 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:06.222323 30734 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:48:06.223220 30734 webserver.cc:460] Webserver started at http://127.4.217.3:36403/ using document root <none> and password file <none>
I20260808 04:48:06.223765 30734 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:06.223863 30734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:06.224089 30734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:48:06.226114 30734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "897ff2411dc341b691f2c22507a93df6"
format_stamp: "Formatted at 2026-08-08 04:48:06 on dist-test-slave-2kcd"
server_key: "4795a2a72fbe8b1762f59442d5c405cf"
server_key_iv: "c953516038645fac726527f7f8d9c1ef"
server_key_version: "encryptionkey@0"
I20260808 04:48:06.226665 30734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "897ff2411dc341b691f2c22507a93df6"
format_stamp: "Formatted at 2026-08-08 04:48:06 on dist-test-slave-2kcd"
server_key: "4795a2a72fbe8b1762f59442d5c405cf"
server_key_iv: "c953516038645fac726527f7f8d9c1ef"
server_key_version: "encryptionkey@0"
I20260808 04:48:06.230404 30734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:48:06.232738 30752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:06.233978 30734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:48:06.234107 30734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "897ff2411dc341b691f2c22507a93df6"
format_stamp: "Formatted at 2026-08-08 04:48:06 on dist-test-slave-2kcd"
server_key: "4795a2a72fbe8b1762f59442d5c405cf"
server_key_iv: "c953516038645fac726527f7f8d9c1ef"
server_key_version: "encryptionkey@0"
I20260808 04:48:06.234210 30734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:06.258844 30734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:06.261899 30734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:06.262100 30734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:06.262713 30734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:48:06.263597 30734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:48:06.263644 30734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:06.263712 30734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:48:06.263743 30734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:06.272289 30734 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:39163
I20260808 04:48:06.272356 30954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:39163 every 8 connection(s)
I20260808 04:48:06.273239 30734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:48:06.280251 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 30734
I20260808 04:48:06.280436 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:48:06.280723 4964 external_mini_cluster.cc:1428] Setting key 6dbf888d0594a13d48dfbe68ffee2fe5
Aug 08 04:48:06 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:48:06.287114 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.274996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:47219 (local address 127.4.217.62:35027)
0808 04:48:06.275138 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:06.275142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:06.275984 (+ 842us) server_negotiation.cc:407] Connection header received
0808 04:48:06.277378 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:06.277381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:06.277437 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:06.277547 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:06.278643 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.279175 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.280301 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.280603 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.280710 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.283105 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:06.283108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:06.283111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:06.283133 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:06.284836 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:06.285432 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:06.285437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:06.285439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:06.285531 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:06.285918 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:06.285921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:06.285942 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:06.286144 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:06.286223 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:06.286807 (+ 584us) server_negotiation.cc:299] Negotiation successful
0808 04:48:06.286947 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:48:06.287873 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.275366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:47219)
0808 04:48:06.275828 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:06.275880 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:06.277170 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:06.277712 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:06.277719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:06.278053 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:48:06.278500 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.278513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.279303 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.279312 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.280173 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.280182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.280750 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.280755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.280963 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.281506 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:48:06.281522 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:48:06.282962 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:48:06.285006 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:06.285013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:06.285027 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:06.285317 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:06.285626 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:06.285630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:06.285632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:06.285810 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:06.286251 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:48:06.286257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:48:06.286544 (+ 287us) client_negotiation.cc:769] Sending connection context
0808 04:48:06.286733 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 04:48:06.287046 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":120,"threads_started":1}
I20260808 04:48:06.288998 30955 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35027
I20260808 04:48:06.289242 30955 heartbeater.cc:464] Registering TS with master...
I20260808 04:48:06.289709 30955 heartbeater.cc:511] Master 127.4.217.62:35027 requested a full tablet report, sending...
I20260808 04:48:06.290696 30184 ts_manager.cc:194] Registered new tserver with Master: 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163)
I20260808 04:48:06.291255 30184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:47219
I20260808 04:48:06.294414 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:48:06.303725 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.296349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.4.217.62:35027)
0808 04:48:06.296493 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:06.296498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:06.296676 (+ 178us) server_negotiation.cc:407] Connection header received
0808 04:48:06.296837 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:06.296840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:06.296889 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:06.296978 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:06.297674 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.298173 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.299019 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.299248 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.299356 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.300633 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:06.300637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:06.300640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:06.300662 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:06.302007 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:06.302430 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:06.302434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:06.302435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:06.302522 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:06.302795 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:06.302800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:06.302801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:06.303020 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:06.303200 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:06.303440 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 04:48:06.303533 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:48:06.306484 30184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:48:06.309551 30184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:48:06.325157 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.319016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:34085 (local address 127.0.0.1:53178)
0808 04:48:06.320226 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:06.320240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:06.320446 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:06.321135 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:06.321137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:06.321151 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:06.321216 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.321220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.322221 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.322232 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.323544 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.323554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.324596 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.324600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.324700 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.324736 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:48:06.324875 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 04:48:06.324996 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"mutex_wait_us":92,"thread_start_us":98,"threads_started":1}
I20260808 04:48:06.325170 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.319494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.4.217.2:34085)
0808 04:48:06.320089 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:06.320100 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:06.320369 (+ 269us) server_negotiation.cc:407] Connection header received
0808 04:48:06.320649 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:06.320656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:06.320869 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:06.320995 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:06.321339 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.322086 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.323671 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.324459 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.324578 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.324727 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:06.324916 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 04:48:06.325015 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":150,"threads_started":1}
I20260808 04:48:06.325541 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.319483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.4.217.1:39389)
0808 04:48:06.319839 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:06.319855 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:06.319877 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:48:06.319977 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:06.319984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:06.320190 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:06.320356 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:06.320923 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.322081 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.323956 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.324485 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.324608 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.324749 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:06.325301 (+ 552us) server_negotiation.cc:299] Negotiation successful
0808 04:48:06.325411 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":66,"threads_started":1}
I20260808 04:48:06.325649 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.318635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:39163 (local address 127.0.0.1:50468)
0808 04:48:06.319514 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:06.319569 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:06.319717 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:06.320502 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:06.320509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:06.320538 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:06.320687 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.320692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.322265 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.322269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.323794 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.323806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.325080 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.325084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.325172 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.325207 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:48:06.325437 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 04:48:06.325516 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":116,"spinlock_wait_cycles":3456,"thread_start_us":114,"threads_started":1}
I20260808 04:48:06.325709 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.318775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.4.217.3:39163)
0808 04:48:06.319269 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:06.319283 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:06.319659 (+ 376us) server_negotiation.cc:407] Connection header received
0808 04:48:06.319888 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:06.319893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:06.320128 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:06.320264 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:06.320827 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.322090 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.323931 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.324727 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.324888 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.325022 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:06.325471 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 04:48:06.325570 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":127,"threads_started":1}
I20260808 04:48:06.325201 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.318751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:39389 (local address 127.0.0.1:34860)
0808 04:48:06.319514 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:06.319569 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:06.319696 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:06.320537 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:06.320539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:06.320556 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:06.320718 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.320725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.322265 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.322269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.323794 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.323806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.324831 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.324838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.324946 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.324967 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:48:06.325033 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 04:48:06.325086 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":198,"spinlock_wait_cycles":20096,"thread_start_us":79,"threads_started":1}
I20260808 04:48:06.328429 30602 tablet_service.cc:1467] Processing CreateTablet for tablet 01cee951fc5149c1a421025612cc522f (DEFAULT_TABLE table=test-table [id=3946a99eeb9c42009683ce86fee3c9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:48:06.328428 30839 tablet_service.cc:1467] Processing CreateTablet for tablet 01cee951fc5149c1a421025612cc522f (DEFAULT_TABLE table=test-table [id=3946a99eeb9c42009683ce86fee3c9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:48:06.329530 30602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01cee951fc5149c1a421025612cc522f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:48:06.329543 30839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01cee951fc5149c1a421025612cc522f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:48:06.331322 30376 tablet_service.cc:1467] Processing CreateTablet for tablet 01cee951fc5149c1a421025612cc522f (DEFAULT_TABLE table=test-table [id=3946a99eeb9c42009683ce86fee3c9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:48:06.332338 30376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01cee951fc5149c1a421025612cc522f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:48:06.335803 30986 tablet_bootstrap.cc:492] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: Bootstrap starting.
I20260808 04:48:06.336472 30988 tablet_bootstrap.cc:492] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: Bootstrap starting.
I20260808 04:48:06.339118 30986 tablet_bootstrap.cc:654] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: Neither blocks nor log segments found. Creating new log.
I20260808 04:48:06.339118 30988 tablet_bootstrap.cc:654] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: Neither blocks nor log segments found. Creating new log.
I20260808 04:48:06.339218 30990 tablet_bootstrap.cc:492] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Bootstrap starting.
I20260808 04:48:06.340067 30988 log.cc:824] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:48:06.340067 30986 log.cc:824] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:48:06.341468 30990 tablet_bootstrap.cc:654] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Neither blocks nor log segments found. Creating new log.
I20260808 04:48:06.342096 30986 tablet_bootstrap.cc:492] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: No bootstrap required, opened a new log
I20260808 04:48:06.342208 30988 tablet_bootstrap.cc:492] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: No bootstrap required, opened a new log
I20260808 04:48:06.342331 30986 ts_tablet_manager.cc:1397] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:48:06.342403 30988 ts_tablet_manager.cc:1397] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:48:06.342712 30990 log.cc:824] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Log is configured to *not* fsync() on all Append() calls
I20260808 04:48:06.344645 30990 tablet_bootstrap.cc:492] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: No bootstrap required, opened a new log
I20260808 04:48:06.344841 30990 ts_tablet_manager.cc:1397] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:48:06.345448 30986 raft_consensus.cc:348] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:06.345664 30986 raft_consensus.cc:374] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:48:06.345697 30986 raft_consensus.cc:729] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897ff2411dc341b691f2c22507a93df6, State: Initialized, Role: FOLLOWER
I20260808 04:48:06.346096 30986 consensus_queue.cc:260] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:06.346107 30988 raft_consensus.cc:348] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:06.346355 30988 raft_consensus.cc:374] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:48:06.346405 30988 raft_consensus.cc:729] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed55f3ed3764245a63367fb6b6651c1, State: Initialized, Role: FOLLOWER
I20260808 04:48:06.346690 30955 heartbeater.cc:503] Master 127.4.217.62:35027 was elected leader, sending a full tablet report...
I20260808 04:48:06.347035 30986 ts_tablet_manager.cc:1428] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260808 04:48:06.347728 30988 consensus_queue.cc:260] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:06.348405 30727 heartbeater.cc:503] Master 127.4.217.62:35027 was elected leader, sending a full tablet report...
I20260808 04:48:06.348702 30988 ts_tablet_manager.cc:1428] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:48:06.348635 30990 raft_consensus.cc:348] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:06.348987 30990 raft_consensus.cc:374] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:48:06.349069 30990 raft_consensus.cc:729] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0aabe8e1e94d8ab1062bd0163ca555, State: Initialized, Role: FOLLOWER
I20260808 04:48:06.349467 30990 consensus_queue.cc:260] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:06.350380 30990 ts_tablet_manager.cc:1428] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:48:06.350767 30494 heartbeater.cc:503] Master 127.4.217.62:35027 was elected leader, sending a full tablet report...
I20260808 04:48:06.380156 30997 raft_consensus.cc:482] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:48:06.380354 30997 raft_consensus.cc:504] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:06.381585 30997 leader_election.cc:288] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163), 6b0aabe8e1e94d8ab1062bd0163ca555 (127.4.217.1:39389)
I20260808 04:48:06.385804 30730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.381911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:39163 (local address 127.4.217.2:36499)
0808 04:48:06.382159 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:06.382176 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:06.382362 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:06.382787 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:06.382792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:06.382821 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:06.383032 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.383043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.383830 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.383835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.384640 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.384650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.385356 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.385359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.385468 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.385490 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:48:06.385588 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:48:06.385662 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"spinlock_wait_cycles":9984}
I20260808 04:48:06.385804 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.382042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:36499 (local address 127.4.217.3:39163)
0808 04:48:06.382183 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:06.382191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:06.382251 (+ 60us) server_negotiation.cc:407] Connection header received
0808 04:48:06.382531 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:06.382556 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:06.382637 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:06.382728 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:06.383181 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.383708 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.384811 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.385227 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.385352 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.385430 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:06.385604 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 04:48:06.385677 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:48:06.386466 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.381911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:39389 (local address 127.4.217.2:45517)
0808 04:48:06.382759 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:06.382775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:06.382926 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:06.383250 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:06.383253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:06.383272 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:06.383414 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.383420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.384202 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.384206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.385103 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:06.385114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.385818 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.385823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:06.385930 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.385959 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:48:06.386274 (+ 315us) client_negotiation.cc:240] Negotiation successful
0808 04:48:06.386346 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":290,"spinlock_wait_cycles":2560,"thread_start_us":147,"threads_started":1}
I20260808 04:48:06.386510 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.382211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:45517 (local address 127.4.217.1:39389)
0808 04:48:06.382409 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:06.382416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:06.382870 (+ 454us) server_negotiation.cc:407] Connection header received
0808 04:48:06.383085 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:06.383088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:06.383137 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:06.383225 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:06.383543 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.384067 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.385244 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.385671 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.385804 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.385873 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:06.386311 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 04:48:06.386397 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:48:06.386617 30877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cee951fc5149c1a421025612cc522f" candidate_uuid: "8ed55f3ed3764245a63367fb6b6651c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897ff2411dc341b691f2c22507a93df6" is_pre_election: true
I20260808 04:48:06.386960 30877 raft_consensus.cc:2454] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed55f3ed3764245a63367fb6b6651c1 in term 0.
I20260808 04:48:06.387138 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cee951fc5149c1a421025612cc522f" candidate_uuid: "8ed55f3ed3764245a63367fb6b6651c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" is_pre_election: true
I20260808 04:48:06.387462 30410 raft_consensus.cc:2454] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed55f3ed3764245a63367fb6b6651c1 in term 0.
I20260808 04:48:06.387519 30538 leader_election.cc:302] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 897ff2411dc341b691f2c22507a93df6, 8ed55f3ed3764245a63367fb6b6651c1; no voters:
I20260808 04:48:06.387928 30997 raft_consensus.cc:2781] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:48:06.388042 30997 raft_consensus.cc:482] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:48:06.388106 30997 raft_consensus.cc:3037] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:48:06.389292 30997 raft_consensus.cc:504] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:06.389668 30997 leader_election.cc:288] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [CANDIDATE]: Term 1 election: Requested vote from peers 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163), 6b0aabe8e1e94d8ab1062bd0163ca555 (127.4.217.1:39389)
I20260808 04:48:06.390074 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cee951fc5149c1a421025612cc522f" candidate_uuid: "8ed55f3ed3764245a63367fb6b6651c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555"
I20260808 04:48:06.390074 30877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cee951fc5149c1a421025612cc522f" candidate_uuid: "8ed55f3ed3764245a63367fb6b6651c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897ff2411dc341b691f2c22507a93df6"
I20260808 04:48:06.390210 30877 raft_consensus.cc:3037] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:48:06.390210 30410 raft_consensus.cc:3037] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:48:06.391566 30877 raft_consensus.cc:2454] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed55f3ed3764245a63367fb6b6651c1 in term 1.
I20260808 04:48:06.391561 30410 raft_consensus.cc:2454] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed55f3ed3764245a63367fb6b6651c1 in term 1.
I20260808 04:48:06.391922 30534 leader_election.cc:302] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b0aabe8e1e94d8ab1062bd0163ca555, 8ed55f3ed3764245a63367fb6b6651c1; no voters:
I20260808 04:48:06.392098 30997 raft_consensus.cc:2781] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:48:06.392326 30997 raft_consensus.cc:686] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed55f3ed3764245a63367fb6b6651c1, State: Running, Role: LEADER
I20260808 04:48:06.392681 30997 consensus_queue.cc:237] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:06.396210 30180 catalog_manager.cc:5495] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "8ed55f3ed3764245a63367fb6b6651c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } health_report { overall_health: HEALTHY } } }
I20260808 04:48:06.422673 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.419129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.4.217.2:34085)
0808 04:48:06.419269 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:06.419273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:06.419287 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:48:06.419452 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:06.419454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:06.419510 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:06.419603 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:48:06.419828 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.420326 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.421258 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:06.421483 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:06.421574 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:06.421834 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:48:06.422242 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:48:06.422339 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:06.422420 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 04:48:06.422470 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:48:06.428596 30877 raft_consensus.cc:1261] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [term 1 FOLLOWER]: Refusing update from remote peer 8ed55f3ed3764245a63367fb6b6651c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:48:06.428596 30410 raft_consensus.cc:1261] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Refusing update from remote peer 8ed55f3ed3764245a63367fb6b6651c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:48:06.429373 31001 consensus_queue.cc:1035] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:48:06.429559 30997 consensus_queue.cc:1035] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:48:06.437181 31005 mvcc.cc:205] Tried to move back new op lower bound from 7316129736404262912 to 7316129736267378688. Current Snapshot: MvccSnapshot[applied={T|T < 7316129736404262912}]
I20260808 04:48:06.445137 30184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:48:06.445283 30184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:48:06.447597 30184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Sending DeleteTablet for 3 replicas of tablet 01cee951fc5149c1a421025612cc522f
I20260808 04:48:06.448357 30376 tablet_service.cc:1514] Processing DeleteTablet for tablet 01cee951fc5149c1a421025612cc522f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:48:06 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:34860
I20260808 04:48:06.448479 30839 tablet_service.cc:1514] Processing DeleteTablet for tablet 01cee951fc5149c1a421025612cc522f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:48:06 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:50468
I20260808 04:48:06.448504 30602 tablet_service.cc:1514] Processing DeleteTablet for tablet 01cee951fc5149c1a421025612cc522f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:48:06 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:53178
I20260808 04:48:06.449127 31022 tablet_replica.cc:330] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: stopping tablet replica
I20260808 04:48:06.449160 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 30281
I20260808 04:48:06.455665 31022 raft_consensus.cc:2229] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:48:06.456049 31022 raft_consensus.cc:2258] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:48:06.460980 31025 tablet_replica.cc:330] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: stopping tablet replica
I20260808 04:48:06.461042 31022 ts_tablet_manager.cc:1905] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:48:06.461328 31025 raft_consensus.cc:2229] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:48:06.461805 31025 raft_consensus.cc:2258] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:48:06.463274 31025 ts_tablet_manager.cc:1905] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:48:06.464988 31022 ts_tablet_manager.cc:1918] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:48:06.465142 31022 log.cc:1196] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/01cee951fc5149c1a421025612cc522f
I20260808 04:48:06.465531 31022 ts_tablet_manager.cc:1939] T 01cee951fc5149c1a421025612cc522f P 897ff2411dc341b691f2c22507a93df6: Deleting consensus metadata
I20260808 04:48:06.466015 31025 ts_tablet_manager.cc:1918] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:48:06.466147 31025 log.cc:1196] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/01cee951fc5149c1a421025612cc522f
I20260808 04:48:06.466532 31025 ts_tablet_manager.cc:1939] T 01cee951fc5149c1a421025612cc522f P 8ed55f3ed3764245a63367fb6b6651c1: Deleting consensus metadata
I20260808 04:48:06.466997 30163 catalog_manager.cc:4841] TS 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163): tablet 01cee951fc5149c1a421025612cc522f (table test-table [id=3946a99eeb9c42009683ce86fee3c9ce]) successfully deleted
I20260808 04:48:06.467641 30157 catalog_manager.cc:4841] TS 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085): tablet 01cee951fc5149c1a421025612cc522f (table test-table [id=3946a99eeb9c42009683ce86fee3c9ce]) successfully deleted
W20260808 04:48:06.468120 30159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:39389 (error 108)
I20260808 04:48:06.469158 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 30507
I20260808 04:48:06.469272 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:06.468944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:39389 (local address 127.0.0.1:34870)
0808 04:48:06.469123 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:06.469187 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:39389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260808 04:48:06.469534 30159 catalog_manager.cc:4584] TS 6b0aabe8e1e94d8ab1062bd0163ca555 (127.4.217.1:39389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01cee951fc5149c1a421025612cc522f: Network error: Client connection negotiation failed: client connection to 127.4.217.1:39389: connect: Connection refused (error 111)
I20260808 04:48:06.486470 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 30734
I20260808 04:48:06.499884 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 30117
I20260808 04:48:26.512852 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35027
--webserver_interface=127.4.217.62
--webserver_port=33119
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:48:26.628650 31027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:26.629026 31027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:26.629086 31027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:26.633100 31027 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:48:26.633155 31027 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:48:26.633174 31027 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:48:26.633191 31027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:48:26.633262 31027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:26.633302 31027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:26.633353 31027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:48:26.633391 31027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:48:26.638710 31027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=33119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:26.639791 31027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:26.640905 31027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:26.655903 31039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:26.655956 31038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:26.655905 31041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:26.655875 31027 server_base.cc:1034] running on GCE node
I20260808 04:48:26.656909 31027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:26.658092 31027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:26.659298 31027 hybrid_clock.cc:648] HybridClock initialized: now 1786164506659279 us; error 31 us; skew 500 ppm
Aug 08 04:48:26 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164506, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:26.666339 31027 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:48:26.667296 31027 webserver.cc:460] Webserver started at http://127.4.217.62:33119/ using document root <none> and password file <none>
I20260808 04:48:26.667868 31027 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:26.667944 31027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:26.671993 31027 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:48:26.674149 31051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:26.675374 31027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260808 04:48:26.675518 31027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "5fd3c6afdeb6491b8192fe3faea7acfd"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "b50c9600b80820c785102dcb3bc363d3"
server_key_iv: "9a9e802623378df4f5d1ab383afc95b3"
server_key_version: "encryptionkey@0"
I20260808 04:48:26.676110 31027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:26.685352 31027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:26.688495 31027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:26.688711 31027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:26.696363 31027 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:35027
I20260808 04:48:26.696672 31147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:35027 every 8 connection(s)
I20260808 04:48:26.697468 31027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:48:26.700482 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 31027
I20260808 04:48:26.701049 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:39389
--local_ip_for_outbound_sockets=127.4.217.1
--tserver_master_addrs=127.4.217.62:35027
--webserver_port=41057
--webserver_interface=127.4.217.1
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:48:26.702440 31148 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:48:26.704430 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Bootstrap starting.
I20260808 04:48:26.722043 31148 log.cc:824] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:48:26.731683 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:48:26.732100 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Bootstrap complete.
I20260808 04:48:26.736352 31148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:26.736747 31148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd3c6afdeb6491b8192fe3faea7acfd, State: Initialized, Role: FOLLOWER
I20260808 04:48:26.737321 31148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:26.737461 31148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:48:26.737560 31148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:48:26.737695 31148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:48:26.739645 31148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:26.740000 31148 leader_election.cc:302] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fd3c6afdeb6491b8192fe3faea7acfd; no voters:
I20260808 04:48:26.740376 31148 leader_election.cc:288] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:48:26.740463 31153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:48:26.740664 31153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 2 LEADER]: Becoming Leader. State: Replica: 5fd3c6afdeb6491b8192fe3faea7acfd, State: Running, Role: LEADER
I20260808 04:48:26.741050 31153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:26.741677 31148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:48:26.742524 31155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } } }
I20260808 04:48:26.742653 31155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: This master's current role is: LEADER
I20260808 04:48:26.743027 31156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd3c6afdeb6491b8192fe3faea7acfd. Latest consensus state: current_term: 2 leader_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } } }
I20260808 04:48:26.743165 31156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: This master's current role is: LEADER
I20260808 04:48:26.743489 31162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:48:26.748497 31162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:48:26.749228 31162 catalog_manager.cc:1259] Loaded cluster ID: 11ad376ca1684b43850e7c657afcd709
I20260808 04:48:26.749289 31162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:48:26.751690 31162 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:48:26.752390 31162 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Loaded TSK: 0
I20260808 04:48:26.767220 31162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Generated new TSK 1
I20260808 04:48:26.767488 31162 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:48:26.823678 31150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:26.823933 31150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:26.823977 31150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:26.828014 31150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:26.828076 31150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:26.828161 31150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:48:26.833386 31150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:39389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=41057
--tserver_master_addrs=127.4.217.62:35027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:26.834378 31150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:26.835386 31150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:26.845099 31190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:26.845132 31189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:26.845433 31150 server_base.cc:1034] running on GCE node
W20260808 04:48:26.845463 31195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:26.845911 31150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:26.846665 31150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:26.847879 31150 hybrid_clock.cc:648] HybridClock initialized: now 1786164506847855 us; error 32 us; skew 500 ppm
Aug 08 04:48:26 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164506, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:26.851166 31150 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:48:26.852134 31150 webserver.cc:460] Webserver started at http://127.4.217.1:41057/ using document root <none> and password file <none>
I20260808 04:48:26.852696 31150 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:26.852830 31150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:26.856734 31150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:48:26.859067 31209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:26.860349 31150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:48:26.860495 31150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "6b0aabe8e1e94d8ab1062bd0163ca555"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "dcd91eb509eac3d658d82aafd2de0fde"
server_key_iv: "65738779e57b75f7e43cfbcbd51340a6"
server_key_version: "encryptionkey@0"
I20260808 04:48:26.861091 31150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:26.878327 31150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:26.882212 31150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:26.882448 31150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:26.883257 31150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:48:26.884636 31220 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:48:26.888595 31150 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:48:26.888657 31150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260808 04:48:26.888725 31150 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:48:26.890342 31150 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:48:26.890400 31150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 04:48:26.890657 31220 tablet_bootstrap.cc:492] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Bootstrap starting.
I20260808 04:48:26.901530 31150 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:39389
I20260808 04:48:26.901588 31415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:39389 every 8 connection(s)
I20260808 04:48:26.902494 31150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:48:26.909075 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 31150
I20260808 04:48:26.909626 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:34085
--local_ip_for_outbound_sockets=127.4.217.2
--tserver_master_addrs=127.4.217.62:35027
--webserver_port=40477
--webserver_interface=127.4.217.2
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:48:26.912194 31220 log.cc:824] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:48:26 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164506, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:48:26.920064 31220 tablet_bootstrap.cc:492] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:48:26.920432 31220 tablet_bootstrap.cc:492] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Bootstrap complete.
I20260808 04:48:26.922040 31220 ts_tablet_manager.cc:1397] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.008s
I20260808 04:48:26.922530 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:26.904704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:45291)
0808 04:48:26.905331 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:26.905402 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:26.906449 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:26.907581 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:26.907589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:26.908058 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:48:26.908896 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:26.908917 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:26.910822 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:26.910825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:26.912078 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:26.912089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:26.913204 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:26.913208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:26.913397 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:26.914109 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:48:26.914129 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:48:26.915976 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:48:26.918854 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:26.918859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:26.918871 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:26.919199 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:26.919676 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:26.919679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:26.919681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:26.919846 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:26.920678 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:48:26.920683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:48:26.921027 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 04:48:26.921225 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 04:48:26.921569 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":154,"threads_started":1}
I20260808 04:48:26.922696 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:26.904831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:45291 (local address 127.4.217.62:35027)
0808 04:48:26.905329 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:26.905351 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:26.905557 (+ 206us) server_negotiation.cc:407] Connection header received
0808 04:48:26.906873 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:26.906883 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:26.907174 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:26.907448 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:26.909105 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:26.910631 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:26.912228 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:26.913045 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:26.913257 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:26.916209 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:26.916219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:26.916240 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:26.916283 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:26.918703 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:26.919347 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:26.919357 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:26.919365 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:26.919545 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:26.919970 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:26.919977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:26.919981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:26.920523 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:26.920703 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:26.921408 (+ 705us) server_negotiation.cc:299] Negotiation successful
0808 04:48:26.921746 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":124,"threads_started":1}
I20260808 04:48:26.925060 31416 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35027
I20260808 04:48:26.925324 31416 heartbeater.cc:464] Registering TS with master...
I20260808 04:48:26.925022 31220 raft_consensus.cc:348] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:26.925560 31220 raft_consensus.cc:729] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0aabe8e1e94d8ab1062bd0163ca555, State: Initialized, Role: FOLLOWER
I20260808 04:48:26.925882 31416 heartbeater.cc:511] Master 127.4.217.62:35027 requested a full tablet report, sending...
I20260808 04:48:26.926087 31220 consensus_queue.cc:260] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } } peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } }
I20260808 04:48:26.926998 31220 ts_tablet_manager.cc:1428] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:48:26.928592 31080 ts_manager.cc:194] Registered new tserver with Master: 6b0aabe8e1e94d8ab1062bd0163ca555 (127.4.217.1:39389)
I20260808 04:48:26.931789 31080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:45291
I20260808 04:48:26.934567 31416 heartbeater.cc:503] Master 127.4.217.62:35027 was elected leader, sending a full tablet report...
Aug 08 04:48:26 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164506, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.1@KRBTEST.COM
I20260808 04:48:26.943707 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:26.931297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:39389 (local address 127.0.0.1:56680)
0808 04:48:26.931841 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:26.931884 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:26.932024 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:26.932763 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:26.932770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:26.935125 (+ 2355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:48:26.935259 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:26.935265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:26.936238 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:26.936262 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:26.936973 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:26.936984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:26.937554 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:26.937558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:26.937677 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:26.938213 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:48:26.938225 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:48:26.939528 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:48:26.941793 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:26.941801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:26.941805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:26.942094 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:26.942474 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:26.942476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:26.942478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:26.942555 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:26.943102 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:48:26.943111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:48:26.943235 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 04:48:26.943414 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 04:48:26.943586 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":124,"threads_started":1}
I20260808 04:48:26.943907 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:26.931379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.4.217.1:39389)
0808 04:48:26.932279 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:26.932290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:26.932310 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:48:26.932372 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:26.932376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:26.932568 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:26.932694 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:26.935375 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:26.936102 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:26.937103 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:26.937431 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:26.937552 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:26.939674 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:26.939686 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:26.939698 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:26.939728 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:26.941674 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:26.942244 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:26.942250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:26.942252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:26.942370 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:26.942652 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:26.942655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:26.942657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:26.942963 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:26.943140 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:26.943559 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 04:48:26.943731 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":107,"threads_started":1}
I20260808 04:48:26.944882 31292 tablet_service.cc:1514] Processing DeleteTablet for tablet 01cee951fc5149c1a421025612cc522f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:48:06 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:56680
I20260808 04:48:26.945675 31431 tablet_replica.cc:330] stopping tablet replica
I20260808 04:48:26.945958 31431 raft_consensus.cc:2229] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:48:26.946192 31431 raft_consensus.cc:2258] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:48:26.947098 31431 ts_tablet_manager.cc:1905] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:48:26.950528 31431 ts_tablet_manager.cc:1918] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:48:26.950608 31431 log.cc:1196] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal/wals/01cee951fc5149c1a421025612cc522f
I20260808 04:48:26.950865 31431 ts_tablet_manager.cc:1939] T 01cee951fc5149c1a421025612cc522f P 6b0aabe8e1e94d8ab1062bd0163ca555: Deleting consensus metadata
I20260808 04:48:26.952498 31053 catalog_manager.cc:4841] TS 6b0aabe8e1e94d8ab1062bd0163ca555 (127.4.217.1:39389): tablet 01cee951fc5149c1a421025612cc522f (table test-table [id=3946a99eeb9c42009683ce86fee3c9ce]) successfully deleted
W20260808 04:48:27.035928 31423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:27.036202 31423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:27.036283 31423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:27.040306 31423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:27.040391 31423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:27.040482 31423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:48:27.045873 31423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:34085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=40477
--tserver_master_addrs=127.4.217.62:35027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:27.046944 31423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:27.047966 31423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:27.057902 31443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:27.057914 31440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:27.057925 31441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:27.058107 31423 server_base.cc:1034] running on GCE node
I20260808 04:48:27.058619 31423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:27.059273 31423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:27.060465 31423 hybrid_clock.cc:648] HybridClock initialized: now 1786164507060451 us; error 27 us; skew 500 ppm
Aug 08 04:48:27 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:27.063283 31423 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:48:27.064221 31423 webserver.cc:460] Webserver started at http://127.4.217.2:40477/ using document root <none> and password file <none>
I20260808 04:48:27.064913 31423 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:27.065011 31423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:27.068660 31423 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:48:27.070801 31456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:27.072057 31423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:48:27.072149 31423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "8ed55f3ed3764245a63367fb6b6651c1"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "4d2833defded0a6f99226e2c70f46703"
server_key_iv: "4393d9e270ee4183ccdb69075b848833"
server_key_version: "encryptionkey@0"
I20260808 04:48:27.072633 31423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:27.093602 31423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:27.096926 31423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:27.097085 31423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:27.097608 31423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:48:27.098379 31423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:48:27.098426 31423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:27.098467 31423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:48:27.098483 31423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:27.107630 31423 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:34085
I20260808 04:48:27.107681 31647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:34085 every 8 connection(s)
I20260808 04:48:27.108615 31423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
Aug 08 04:48:27 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:48:27.117976 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 31423
I20260808 04:48:27.118415 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:39163
--local_ip_for_outbound_sockets=127.4.217.3
--tserver_master_addrs=127.4.217.62:35027
--webserver_port=36403
--webserver_interface=127.4.217.3
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:48:27.121925 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.110499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:34283 (local address 127.4.217.62:35027)
0808 04:48:27.110659 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.110664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.111528 (+ 864us) server_negotiation.cc:407] Connection header received
0808 04:48:27.112757 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.112760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.112851 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.112936 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:27.114059 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.114600 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.115716 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.115934 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.116009 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.118383 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:27.118387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:27.118389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:27.118413 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:27.119891 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:27.120398 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:27.120402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:27.120403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:27.120491 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:27.120862 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:27.120866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:27.120867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:27.121044 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:27.121157 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.121625 (+ 468us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.121751 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:48:27.122658 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.110897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:34283)
0808 04:48:27.111375 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:27.111416 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:27.112563 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:27.113115 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:27.113122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:27.113468 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:48:27.113926 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.113940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.114712 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.114715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.115606 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.115616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.116072 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.116075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.116214 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.116743 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:48:27.116760 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:48:27.118217 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:48:27.120018 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:27.120024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:27.120037 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:27.120290 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:27.120587 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:27.120590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:27.120592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:27.120730 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:27.121157 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:48:27.121165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:48:27.121399 (+ 234us) client_negotiation.cc:769] Sending connection context
0808 04:48:27.121580 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 04:48:27.121877 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":118,"threads_started":1}
I20260808 04:48:27.123847 31648 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35027
I20260808 04:48:27.124096 31648 heartbeater.cc:464] Registering TS with master...
I20260808 04:48:27.124572 31648 heartbeater.cc:511] Master 127.4.217.62:35027 requested a full tablet report, sending...
I20260808 04:48:27.125691 31080 ts_manager.cc:194] Registered new tserver with Master: 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085)
I20260808 04:48:27.126268 31080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:34283
W20260808 04:48:27.232326 31655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:27.232623 31655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:27.232698 31655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:27.236675 31655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:27.236758 31655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:27.236922 31655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:48:27.242472 31655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:39163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=36403
--tserver_master_addrs=127.4.217.62:35027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:27.243682 31655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:27.244707 31655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:27.254733 31664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:27.254757 31665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:27.255010 31669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:27.255056 31655 server_base.cc:1034] running on GCE node
I20260808 04:48:27.255488 31655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:27.256103 31655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:27.257261 31655 hybrid_clock.cc:648] HybridClock initialized: now 1786164507257245 us; error 31 us; skew 500 ppm
Aug 08 04:48:27 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:27.260000 31655 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:48:27.260910 31655 webserver.cc:460] Webserver started at http://127.4.217.3:36403/ using document root <none> and password file <none>
I20260808 04:48:27.261497 31655 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:27.261607 31655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:27.265717 31655 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:48:27.267757 31680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:27.269130 31655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:48:27.269263 31655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "897ff2411dc341b691f2c22507a93df6"
format_stamp: "Formatted at 2026-08-08 04:48:06 on dist-test-slave-2kcd"
server_key: "4795a2a72fbe8b1762f59442d5c405cf"
server_key_iv: "c953516038645fac726527f7f8d9c1ef"
server_key_version: "encryptionkey@0"
I20260808 04:48:27.269806 31655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:27.285089 31655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:27.288064 31655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:27.288265 31655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:27.288932 31655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:48:27.289772 31655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:48:27.289844 31655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:27.289916 31655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:48:27.289966 31655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:27.300104 31655 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:39163
I20260808 04:48:27.300135 31867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:39163 every 8 connection(s)
I20260808 04:48:27.301146 31655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:48:27.306111 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 31655
Aug 08 04:48:27 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:48:27.315570 31873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.306832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.4.217.62:35027)
0808 04:48:27.307064 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.307069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.307257 (+ 188us) server_negotiation.cc:407] Connection header received
0808 04:48:27.307422 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.307430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.307469 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.307566 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:27.308340 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.308957 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.309790 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.310137 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.310246 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.311821 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:27.311828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:27.311831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:27.311864 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:27.313683 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:27.314194 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:27.314197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:27.314198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:27.314272 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:27.314572 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:27.314577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:27.314579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:27.314794 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:27.315004 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.315220 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.315341 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":63,"threads_started":1}
I20260808 04:48:27.316704 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.303005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:37729 (local address 127.4.217.62:35027)
0808 04:48:27.303143 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.303147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.303941 (+ 794us) server_negotiation.cc:407] Connection header received
0808 04:48:27.305313 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.305316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.305381 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.305464 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:27.306943 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.307473 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.309005 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.309248 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.309386 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.312531 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:27.312534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:27.312536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:27.312576 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:27.314350 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:27.314943 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:27.314946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:27.314948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:27.315038 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:27.315514 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:27.315518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:27.315520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:27.315761 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:27.315853 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.316446 (+ 593us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.316559 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:48:27.317623 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.303316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:37729)
0808 04:48:27.303754 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:27.303795 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:27.305092 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:27.305638 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:27.305645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:27.306124 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:48:27.306792 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.306808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.307609 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.307612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.308761 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.308773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.309435 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.309439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.309639 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.310362 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:48:27.310387 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:48:27.312333 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:48:27.314461 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:27.314465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:27.314477 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:27.314825 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:27.315161 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:27.315164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:27.315166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:27.315360 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:27.315875 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:48:27.315881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:48:27.316196 (+ 315us) client_negotiation.cc:769] Sending connection context
0808 04:48:27.316400 (+ 204us) client_negotiation.cc:240] Negotiation successful
0808 04:48:27.316715 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260808 04:48:27.318610 31868 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35027
I20260808 04:48:27.318833 31868 heartbeater.cc:464] Registering TS with master...
I20260808 04:48:27.319252 31868 heartbeater.cc:511] Master 127.4.217.62:35027 requested a full tablet report, sending...
I20260808 04:48:27.320182 31080 ts_manager.cc:194] Registered new tserver with Master: 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163)
I20260808 04:48:27.320672 31080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:37729
I20260808 04:48:27.329388 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:48:27.333174 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.329982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.4.217.62:35027)
0808 04:48:27.330115 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.330119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.330269 (+ 150us) server_negotiation.cc:407] Connection header received
0808 04:48:27.330391 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.330393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.330478 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.330558 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:48:27.330755 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.331259 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.332027 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.332221 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.332294 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.332581 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:48:27.332701 (+ 120us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 04:48:27.333041 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53296: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:48:27.333258 31422 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53296: token expired
W20260808 04:48:27.333640 30959 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.217.62:35027)
I20260808 04:48:27.340960 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.334011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.4.217.62:35027)
0808 04:48:27.334126 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.334132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.334175 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:48:27.334318 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.334321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.334366 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.334445 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:27.335040 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.335514 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.336338 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.336554 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.336657 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.337925 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:27.337929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:27.337932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:27.337956 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:27.339228 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:27.339687 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:27.339690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:27.339692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:27.339777 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:27.340100 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:27.340104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:27.340106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:27.340277 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:27.340423 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.340663 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.340778 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:48:27.343351 31080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:48:27.346709 31080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:48:27.354552 31292 tablet_service.cc:1467] Processing CreateTablet for tablet c4f8db37cf754dd1bf19a3e612184187 (DEFAULT_TABLE table=test-table [id=ad20328b2df34164a7cdef334b26b0be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:48:27.355218 31292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f8db37cf754dd1bf19a3e612184187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:48:27.358168 31220 tablet_bootstrap.cc:492] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Bootstrap starting.
I20260808 04:48:27.358845 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.353427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:34085 (local address 127.0.0.1:37044)
0808 04:48:27.353624 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:27.353640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:27.353712 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:27.354632 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:27.354635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:27.354656 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:27.354742 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.354748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.356016 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.356019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.357244 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.357255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.358381 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.358384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.358533 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.358582 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 04:48:27.358640 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 04:48:27.358723 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 04:48:27.359191 31220 tablet_bootstrap.cc:654] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Neither blocks nor log segments found. Creating new log.
I20260808 04:48:27.359597 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.353593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.4.217.2:34085)
0808 04:48:27.354083 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.354096 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.354122 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:48:27.354223 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.354229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.354461 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.354633 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:27.354825 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.355875 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.357411 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.358226 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.359142 (+ 916us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.359292 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.359346 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.359450 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":114,"threads_started":1}
I20260808 04:48:27.360805 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.353716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:39163 (local address 127.0.0.1:42322)
0808 04:48:27.354199 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:27.354214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:27.354312 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:27.355007 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:27.355010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:27.355028 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:27.355120 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.355126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.356366 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.356372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.357226 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.357236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.358004 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.358008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.360430 (+ 2422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.360456 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:48:27.360540 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:48:27.360620 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":102,"threads_started":1}
I20260808 04:48:27.361135 31220 tablet_bootstrap.cc:492] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: No bootstrap required, opened a new log
I20260808 04:48:27.361231 31220 ts_tablet_manager.cc:1397] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:48:27.361485 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.354105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.4.217.3:39163)
0808 04:48:27.354580 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.354594 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.354616 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:48:27.354683 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.354688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.354879 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.355030 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:27.355261 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.356206 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.357358 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.357875 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.357991 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.358156 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.361228 (+ 3072us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.361337 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":198,"threads_started":1}
I20260808 04:48:27.361719 31220 raft_consensus.cc:348] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:27.361827 31220 raft_consensus.cc:374] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:48:27.361864 31220 raft_consensus.cc:729] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0aabe8e1e94d8ab1062bd0163ca555, State: Initialized, Role: FOLLOWER
I20260808 04:48:27.361984 31220 consensus_queue.cc:260] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:27.362264 31220 ts_tablet_manager.cc:1428] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:48:27.362835 31529 tablet_service.cc:1467] Processing CreateTablet for tablet c4f8db37cf754dd1bf19a3e612184187 (DEFAULT_TABLE table=test-table [id=ad20328b2df34164a7cdef334b26b0be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:48:27.363991 31529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f8db37cf754dd1bf19a3e612184187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:48:27.364008 31752 tablet_service.cc:1467] Processing CreateTablet for tablet c4f8db37cf754dd1bf19a3e612184187 (DEFAULT_TABLE table=test-table [id=ad20328b2df34164a7cdef334b26b0be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:48:27.365039 31752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f8db37cf754dd1bf19a3e612184187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:48:27.367015 31426 raft_consensus.cc:482] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:48:27.367154 31426 raft_consensus.cc:504] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:27.368237 31426 leader_election.cc:288] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085), 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163)
I20260808 04:48:27.373104 31893 tablet_bootstrap.cc:492] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: Bootstrap starting.
I20260808 04:48:27.373142 31892 tablet_bootstrap.cc:492] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: Bootstrap starting.
I20260808 04:48:27.374643 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.368547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:39163 (local address 127.4.217.1:49997)
0808 04:48:27.368722 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:27.368736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:27.368870 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:27.369558 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:27.369563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:27.369588 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:27.369727 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.369734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.371805 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.371809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.373129 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.373141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.374135 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.374139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.374276 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.374323 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 04:48:27.374415 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:48:27.374495 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":77}
I20260808 04:48:27.375344 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.369471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:47693 (local address 127.4.217.2:34085)
0808 04:48:27.370421 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.370425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.370439 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:48:27.370482 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.370487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.370710 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.370829 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:27.371223 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.371952 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.373936 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.374591 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.374680 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.374740 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.375151 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.375223 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872}
I20260808 04:48:27.376017 31891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.369147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:34085 (local address 127.4.217.1:47693)
0808 04:48:27.369436 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:27.369449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:27.369523 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:27.370892 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:27.370895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:27.370918 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:27.371056 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.371065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.372126 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.372129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.373755 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.373768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.374863 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.374868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.374982 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.375009 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:48:27.375677 (+ 668us) client_negotiation.cc:240] Negotiation successful
0808 04:48:27.375777 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260808 04:48:27.376014 31892 tablet_bootstrap.cc:654] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: Neither blocks nor log segments found. Creating new log.
I20260808 04:48:27.376106 31893 tablet_bootstrap.cc:654] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: Neither blocks nor log segments found. Creating new log.
I20260808 04:48:27.376250 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.368670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:49997 (local address 127.4.217.3:39163)
0808 04:48:27.369204 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.369207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.369218 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:48:27.369252 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.369254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.369315 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.369401 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:27.370950 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.371660 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.373318 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.373885 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.374001 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.374074 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.376003 (+ 1929us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.376080 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":130,"threads_started":1}
I20260808 04:48:27.377133 31893 log.cc:824] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:48:27.376893 31776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db37cf754dd1bf19a3e612184187" candidate_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897ff2411dc341b691f2c22507a93df6" is_pre_election: true
I20260808 04:48:27.378022 31892 log.cc:824] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:48:27.376874 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db37cf754dd1bf19a3e612184187" candidate_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed55f3ed3764245a63367fb6b6651c1" is_pre_election: true
W20260808 04:48:27.378860 31214 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163): Illegal state: must be running to vote when last-logged opid is not known
W20260808 04:48:27.379073 31210 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085): Illegal state: must be running to vote when last-logged opid is not known
I20260808 04:48:27.379153 31210 leader_election.cc:302] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0aabe8e1e94d8ab1062bd0163ca555; no voters: 897ff2411dc341b691f2c22507a93df6, 8ed55f3ed3764245a63367fb6b6651c1
I20260808 04:48:27.379387 31893 tablet_bootstrap.cc:492] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: No bootstrap required, opened a new log
I20260808 04:48:27.379419 31426 raft_consensus.cc:2726] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 04:48:27.379645 31893 ts_tablet_manager.cc:1397] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:48:27.380012 31892 tablet_bootstrap.cc:492] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: No bootstrap required, opened a new log
I20260808 04:48:27.380206 31892 ts_tablet_manager.cc:1397] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:48:27.382364 31893 raft_consensus.cc:348] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:27.382557 31893 raft_consensus.cc:374] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:48:27.382614 31893 raft_consensus.cc:729] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897ff2411dc341b691f2c22507a93df6, State: Initialized, Role: FOLLOWER
I20260808 04:48:27.382714 31892 raft_consensus.cc:348] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:27.382956 31892 raft_consensus.cc:374] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:48:27.383023 31892 raft_consensus.cc:729] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed55f3ed3764245a63367fb6b6651c1, State: Initialized, Role: FOLLOWER
I20260808 04:48:27.383118 31893 consensus_queue.cc:260] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:27.383466 31892 consensus_queue.cc:260] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:27.383764 31868 heartbeater.cc:503] Master 127.4.217.62:35027 was elected leader, sending a full tablet report...
I20260808 04:48:27.384169 31893 ts_tablet_manager.cc:1428] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260808 04:48:27.384177 31648 heartbeater.cc:503] Master 127.4.217.62:35027 was elected leader, sending a full tablet report...
I20260808 04:48:27.384477 31892 ts_tablet_manager.cc:1428] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260808 04:48:27.403748 31417 tablet.cc:2369] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:48:27.536142 31901 raft_consensus.cc:482] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:48:27.536404 31901 raft_consensus.cc:504] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:27.537714 31901 leader_election.cc:288] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163), 6b0aabe8e1e94d8ab1062bd0163ca555 (127.4.217.1:39389)
I20260808 04:48:27.542668 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.538048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:39329 (local address 127.4.217.3:39163)
0808 04:48:27.538191 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.538195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.538211 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:48:27.538434 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.538440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.538506 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.538607 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:27.538938 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.539762 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.541407 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.542032 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.542170 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.542242 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.542446 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.542513 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:48:27.542697 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.538413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:53819 (local address 127.4.217.1:39389)
0808 04:48:27.538705 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.538709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.538724 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:48:27.538777 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.538779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.538817 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.538920 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:48:27.539180 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.539965 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.541411 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.542126 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.542218 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.542351 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.542444 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.542513 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260808 04:48:27.542726 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.537979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:39163 (local address 127.4.217.2:39329)
0808 04:48:27.538140 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:27.538158 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:27.538269 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:27.538631 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:27.538634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:27.538677 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:27.538818 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.538825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.539873 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.539876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.541253 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.541266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.542160 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.542162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.542294 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.542341 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 04:48:27.542418 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:48:27.542528 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":120,"spinlock_wait_cycles":25344}
I20260808 04:48:27.542783 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.538126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:39389 (local address 127.4.217.2:53819)
0808 04:48:27.538552 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:27.538567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:27.538683 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:27.538934 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:27.538938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:27.538958 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:48:27.539062 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.539066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.540088 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.540091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.541296 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:27.541306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.542221 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.542224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:27.542327 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.542342 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 04:48:27.542405 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 04:48:27.542527 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":58,"spinlock_wait_cycles":8192,"thread_start_us":157,"threads_started":1}
I20260808 04:48:27.543443 31776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db37cf754dd1bf19a3e612184187" candidate_uuid: "8ed55f3ed3764245a63367fb6b6651c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897ff2411dc341b691f2c22507a93df6" is_pre_election: true
I20260808 04:48:27.543535 31329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db37cf754dd1bf19a3e612184187" candidate_uuid: "8ed55f3ed3764245a63367fb6b6651c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" is_pre_election: true
I20260808 04:48:27.543762 31776 raft_consensus.cc:2454] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed55f3ed3764245a63367fb6b6651c1 in term 0.
I20260808 04:48:27.543778 31329 raft_consensus.cc:2454] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed55f3ed3764245a63367fb6b6651c1 in term 0.
I20260808 04:48:27.544224 31458 leader_election.cc:302] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6b0aabe8e1e94d8ab1062bd0163ca555, 897ff2411dc341b691f2c22507a93df6, 8ed55f3ed3764245a63367fb6b6651c1; no voters:
I20260808 04:48:27.544559 31901 raft_consensus.cc:2781] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:48:27.544646 31901 raft_consensus.cc:482] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:48:27.544677 31901 raft_consensus.cc:3037] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:48:27.545787 31901 raft_consensus.cc:504] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:27.546142 31901 leader_election.cc:288] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [CANDIDATE]: Term 1 election: Requested vote from peers 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163), 6b0aabe8e1e94d8ab1062bd0163ca555 (127.4.217.1:39389)
I20260808 04:48:27.546486 31776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db37cf754dd1bf19a3e612184187" candidate_uuid: "8ed55f3ed3764245a63367fb6b6651c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897ff2411dc341b691f2c22507a93df6"
I20260808 04:48:27.546622 31329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db37cf754dd1bf19a3e612184187" candidate_uuid: "8ed55f3ed3764245a63367fb6b6651c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555"
I20260808 04:48:27.546638 31776 raft_consensus.cc:3037] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:48:27.546757 31329 raft_consensus.cc:3037] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:48:27.547960 31776 raft_consensus.cc:2454] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed55f3ed3764245a63367fb6b6651c1 in term 1.
I20260808 04:48:27.547960 31329 raft_consensus.cc:2454] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed55f3ed3764245a63367fb6b6651c1 in term 1.
I20260808 04:48:27.548290 31461 leader_election.cc:302] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 897ff2411dc341b691f2c22507a93df6, 8ed55f3ed3764245a63367fb6b6651c1; no voters:
I20260808 04:48:27.548502 31901 raft_consensus.cc:2781] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:48:27.548691 31901 raft_consensus.cc:686] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed55f3ed3764245a63367fb6b6651c1, State: Running, Role: LEADER
I20260808 04:48:27.549072 31901 consensus_queue.cc:237] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
W20260808 04:48:27.552033 31869 tablet.cc:2369] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:48:27.552268 31080 catalog_manager.cc:5495] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "8ed55f3ed3764245a63367fb6b6651c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } health_report { overall_health: UNKNOWN } } }
I20260808 04:48:27.599962 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:27.596232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.4.217.2:34085)
0808 04:48:27.596398 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:27.596403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:27.596418 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:48:27.596547 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:27.596550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:27.596607 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:27.596721 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:48:27.597041 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.597552 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.598451 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:27.598667 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:27.598782 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:27.599111 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:48:27.599538 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:48:27.599670 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:27.599756 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 04:48:27.599811 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:48:27.605944 31776 raft_consensus.cc:1261] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [term 1 FOLLOWER]: Refusing update from remote peer 8ed55f3ed3764245a63367fb6b6651c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:48:27.605981 31329 raft_consensus.cc:1261] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Refusing update from remote peer 8ed55f3ed3764245a63367fb6b6651c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:48:27.606664 31901 consensus_queue.cc:1035] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:48:27.606845 31906 consensus_queue.cc:1035] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:48:27.609750 31652 tablet.cc:2369] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:48:27.622381 31080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:48:27.622586 31080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:48:27.624846 31080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Sending DeleteTablet for 3 replicas of tablet c4f8db37cf754dd1bf19a3e612184187
I20260808 04:48:27.625554 31529 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f8db37cf754dd1bf19a3e612184187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:48:27 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:37044
I20260808 04:48:27.625672 31752 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f8db37cf754dd1bf19a3e612184187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:48:27 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260808 04:48:27.625717 31292 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f8db37cf754dd1bf19a3e612184187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:48:27 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:56680
I20260808 04:48:27.626080 31924 tablet_replica.cc:330] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: stopping tablet replica
I20260808 04:48:27.626348 31926 tablet_replica.cc:330] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: stopping tablet replica
I20260808 04:48:27.626395 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 31150
I20260808 04:48:27.626407 31924 raft_consensus.cc:2229] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:48:27.626472 31926 raft_consensus.cc:2229] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:48:27.626590 31926 raft_consensus.cc:2258] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:48:27.626837 31922 tablet_replica.cc:330] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: stopping tablet replica
I20260808 04:48:27.627104 31922 raft_consensus.cc:2229] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:48:27.627393 31922 raft_consensus.cc:2258] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:48:27.633046 31924 raft_consensus.cc:2258] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:48:27.637862 31924 ts_tablet_manager.cc:1905] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:48:27.641528 31924 ts_tablet_manager.cc:1918] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:48:27.641659 31924 log.cc:1196] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/c4f8db37cf754dd1bf19a3e612184187
I20260808 04:48:27.642004 31922 ts_tablet_manager.cc:1905] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:48:27.642494 31924 ts_tablet_manager.cc:1939] T c4f8db37cf754dd1bf19a3e612184187 P 897ff2411dc341b691f2c22507a93df6: Deleting consensus metadata
I20260808 04:48:27.644064 31056 catalog_manager.cc:4841] TS 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163): tablet c4f8db37cf754dd1bf19a3e612184187 (table test-table [id=ad20328b2df34164a7cdef334b26b0be]) successfully deleted
I20260808 04:48:27.645032 31922 ts_tablet_manager.cc:1918] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:48:27.645197 31922 log.cc:1196] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/c4f8db37cf754dd1bf19a3e612184187
W20260808 04:48:27.645356 31053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:39389 (error 108)
I20260808 04:48:27.645643 31922 ts_tablet_manager.cc:1939] T c4f8db37cf754dd1bf19a3e612184187 P 8ed55f3ed3764245a63367fb6b6651c1: Deleting consensus metadata
W20260808 04:48:27.646621 31887 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:48:27.646271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:39389 (local address 127.0.0.1:56694)
0808 04:48:27.646427 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:27.646517 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:39389: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:48:27.646896 31052 catalog_manager.cc:4841] TS 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085): tablet c4f8db37cf754dd1bf19a3e612184187 (table test-table [id=ad20328b2df34164a7cdef334b26b0be]) successfully deleted
W20260808 04:48:27.646772 31053 catalog_manager.cc:4584] TS 6b0aabe8e1e94d8ab1062bd0163ca555 (127.4.217.1:39389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4f8db37cf754dd1bf19a3e612184187: Network error: Client connection negotiation failed: client connection to 127.4.217.1:39389: connect: Connection reset by peer (error 104)
I20260808 04:48:27.647516 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 31423
I20260808 04:48:27.660665 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 31655
I20260808 04:48:27.682683 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 31027
I20260808 04:48:47.697790 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35027
--webserver_interface=127.4.217.62
--webserver_port=33119
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:48:47.816187 31931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:47.816459 31931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:47.816504 31931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:47.820389 31931 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:48:47.820546 31931 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:48:47.820632 31931 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:48:47.820663 31931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:48:47.820686 31931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:47.820715 31931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:47.820739 31931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:48:47.820763 31931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:48:47.826030 31931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=33119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:47.827075 31931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:47.828079 31931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:47.836917 31938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:47.836937 31944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:47.836911 31931 server_base.cc:1034] running on GCE node
W20260808 04:48:47.836890 31939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:47.837764 31931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:47.838810 31931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:47.839982 31931 hybrid_clock.cc:648] HybridClock initialized: now 1786164527839969 us; error 24 us; skew 500 ppm
Aug 08 04:48:47 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164527, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:47.848376 31931 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:48:47.849289 31931 webserver.cc:460] Webserver started at http://127.4.217.62:33119/ using document root <none> and password file <none>
I20260808 04:48:47.849809 31931 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:47.849869 31931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:47.853654 31931 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:48:47.855775 31955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:47.857093 31931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:48:47.857234 31931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "5fd3c6afdeb6491b8192fe3faea7acfd"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "b50c9600b80820c785102dcb3bc363d3"
server_key_iv: "9a9e802623378df4f5d1ab383afc95b3"
server_key_version: "encryptionkey@0"
I20260808 04:48:47.857790 31931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:47.867719 31931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:47.870548 31931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:47.870746 31931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:47.877878 31931 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:35027
I20260808 04:48:47.878170 32068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:35027 every 8 connection(s)
I20260808 04:48:47.878938 31931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:48:47.883152 32069 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:48:47.884753 32069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Bootstrap starting.
I20260808 04:48:47.885800 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 31931
I20260808 04:48:47.886232 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:39389
--local_ip_for_outbound_sockets=127.4.217.1
--tserver_master_addrs=127.4.217.62:35027
--webserver_port=41057
--webserver_interface=127.4.217.1
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:48:47.908043 32069 log.cc:824] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:48:47.922410 32069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:48:47.922829 32069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Bootstrap complete.
I20260808 04:48:47.928197 32069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:47.928687 32069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd3c6afdeb6491b8192fe3faea7acfd, State: Initialized, Role: FOLLOWER
I20260808 04:48:47.929265 32069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:47.929409 32069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:48:47.929479 32069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:48:47.929581 32069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:48:47.932075 32069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:47.932423 32069 leader_election.cc:302] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fd3c6afdeb6491b8192fe3faea7acfd; no voters:
I20260808 04:48:47.932753 32069 leader_election.cc:288] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:48:47.932905 32076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:48:47.933281 32076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [term 3 LEADER]: Becoming Leader. State: Replica: 5fd3c6afdeb6491b8192fe3faea7acfd, State: Running, Role: LEADER
I20260808 04:48:47.933640 32076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } }
I20260808 04:48:47.934087 32069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:48:47.935099 32079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd3c6afdeb6491b8192fe3faea7acfd. Latest consensus state: current_term: 3 leader_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } } }
I20260808 04:48:47.935228 32079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: This master's current role is: LEADER
I20260808 04:48:47.935634 32093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:48:47.935585 32077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd3c6afdeb6491b8192fe3faea7acfd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35027 } } }
I20260808 04:48:47.935742 32077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd [sys.catalog]: This master's current role is: LEADER
I20260808 04:48:47.940760 32093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:48:47.941617 32093 catalog_manager.cc:1259] Loaded cluster ID: 11ad376ca1684b43850e7c657afcd709
I20260808 04:48:47.941682 32093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:48:47.944293 32093 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:48:47.945130 32093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Loaded TSK: 0
I20260808 04:48:47.945217 32093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5fd3c6afdeb6491b8192fe3faea7acfd: Loaded TSK: 1
F20260808 04:48:47.945971 32093 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc36ffce9ad google::LogMessage::Fail() at ??:0
@ 0x7fc36ffd188a google::LogMessage::SendToLog() at ??:0
@ 0x7fc36ffce4de google::LogMessage::Flush() at ??:0
@ 0x7fc36ffcfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc372c9218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc372c8af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc37a9337bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc37a90d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc37a90e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc37a94fdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc37a96e82b std::function<>::operator()() at ??:0
@ 0x7fc37a90db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc37a90ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc37a90b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc37a94f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc3722203d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc372220c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc3722223f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc372211512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc3738556db start_thread at ??:0
@ 0x7fc36eacc61f clone at ??:0
W20260808 04:48:48.035689 32072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:48.035914 32072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:48.035956 32072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:48.040643 32072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:48.040692 32072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:48.040751 32072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:48:48.046653 32072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:39389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=41057
--tserver_master_addrs=127.4.217.62:35027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:48.047853 32072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:48.048969 32072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:48.059140 32144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:48.059176 32143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:48.059566 32146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:48.061022 32072 server_base.cc:1034] running on GCE node
I20260808 04:48:48.061465 32072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:48.062108 32072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:48.063301 32072 hybrid_clock.cc:648] HybridClock initialized: now 1786164528063277 us; error 41 us; skew 500 ppm
Aug 08 04:48:48 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:48.066198 32072 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:48:48.067058 32072 webserver.cc:460] Webserver started at http://127.4.217.1:41057/ using document root <none> and password file <none>
I20260808 04:48:48.067643 32072 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:48.067706 32072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:48.071703 32072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:48:48.074036 32156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:48.075290 32072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:48:48.075436 32072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "6b0aabe8e1e94d8ab1062bd0163ca555"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "dcd91eb509eac3d658d82aafd2de0fde"
server_key_iv: "65738779e57b75f7e43cfbcbd51340a6"
server_key_version: "encryptionkey@0"
I20260808 04:48:48.075968 32072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:48.101837 32072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:48.105140 32072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:48.105360 32072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:48.105959 32072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:48:48.107113 32169 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:48:48.112546 32072 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:48:48.112633 32072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 04:48:48.112721 32072 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:48:48.114482 32072 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:48:48.114549 32072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 04:48:48.114838 32169 tablet_bootstrap.cc:492] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Bootstrap starting.
I20260808 04:48:48.125618 32072 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:39389
I20260808 04:48:48.125855 32348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:39389 every 8 connection(s)
I20260808 04:48:48.126611 32072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:48:48.135043 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 32072
I20260808 04:48:48.135563 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:34085
--local_ip_for_outbound_sockets=127.4.217.2
--tserver_master_addrs=127.4.217.62:35027
--webserver_port=40477
--webserver_interface=127.4.217.2
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:48:48.137792 32169 log.cc:824] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Log is configured to *not* fsync() on all Append() calls
I20260808 04:48:48.146884 32169 tablet_bootstrap.cc:492] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:48:48.147341 32169 tablet_bootstrap.cc:492] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Bootstrap complete.
I20260808 04:48:48.149035 32169 ts_tablet_manager.cc:1397] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.008s
I20260808 04:48:48.152542 32169 raft_consensus.cc:348] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:48.153139 32169 raft_consensus.cc:729] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0aabe8e1e94d8ab1062bd0163ca555, State: Initialized, Role: FOLLOWER
I20260808 04:48:48.153679 32169 consensus_queue.cc:260] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:48.154827 32169 ts_tablet_manager.cc:1428] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260808 04:48:48.202842 32355 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:48:48.128547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:41289)
0808 04:48:48.129065 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:48.129119 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:48.129936 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:48.201800 (+ 71864us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":174,"threads_started":1}
W20260808 04:48:48.203580 32353 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:35027 (0 consecutive failures): Network error: Failed to ping master at 127.4.217.62:35027: Client connection negotiation failed: client connection to 127.4.217.62:35027: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:48:48.204624 32355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:48.204286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:52673)
0808 04:48:48.204464 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:48.204549 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 04:48:48.205447 32355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:48.205248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:56661)
0808 04:48:48.205357 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:48.205401 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:48:48.205706 32353 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:48:48.261392 32359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:48.261680 32359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:48.261770 32359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:48.265841 32359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:48.265928 32359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:48.266026 32359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:48:48.271332 32359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:34085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=40477
--tserver_master_addrs=127.4.217.62:35027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:48.272439 32359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:48.273561 32359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:48:48.283563 32374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:48.283581 32375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:48.283739 32359 server_base.cc:1034] running on GCE node
W20260808 04:48:48.283759 32377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:48.284303 32359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:48.285025 32359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:48.286187 32359 hybrid_clock.cc:648] HybridClock initialized: now 1786164528286174 us; error 24 us; skew 500 ppm
Aug 08 04:48:48 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:48.289050 32359 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:48:48.289893 32359 webserver.cc:460] Webserver started at http://127.4.217.2:40477/ using document root <none> and password file <none>
I20260808 04:48:48.290458 32359 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:48.290521 32359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:48.294266 32359 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:48:48.296370 32388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:48.297567 32359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:48:48.297693 32359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "8ed55f3ed3764245a63367fb6b6651c1"
format_stamp: "Formatted at 2026-08-08 04:48:05 on dist-test-slave-2kcd"
server_key: "4d2833defded0a6f99226e2c70f46703"
server_key_iv: "4393d9e270ee4183ccdb69075b848833"
server_key_version: "encryptionkey@0"
I20260808 04:48:48.298209 32359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:48.309350 32359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:48.312006 32359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:48.312202 32359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:48.312801 32359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:48:48.313673 32359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:48:48.313722 32359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:48.313793 32359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:48:48.313830 32359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:48.323036 32359 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:34085
I20260808 04:48:48.323074 32571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:34085 every 8 connection(s)
I20260808 04:48:48.324303 32359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:48:48.325565 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 32359
I20260808 04:48:48.326062 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:39163
--local_ip_for_outbound_sockets=127.4.217.3
--tserver_master_addrs=127.4.217.62:35027
--webserver_port=36403
--webserver_interface=127.4.217.3
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:48:48.328866 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:48.326622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:56981)
0808 04:48:48.327168 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:48.327318 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":120,"threads_started":1}
W20260808 04:48:48.329368 32573 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:35027 (0 consecutive failures): Network error: Failed to ping master at 127.4.217.62:35027: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
I20260808 04:48:48.330027 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:48.329790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:56627)
0808 04:48:48.329905 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:48.329960 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 04:48:48.330644 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:48.330446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:44673)
0808 04:48:48.330549 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:48.330601 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:48:48.330837 32573 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:48:48.443634 32577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:48:48.443936 32577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:48:48.444025 32577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:48:48.448135 32577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:48:48.448232 32577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:48:48.448326 32577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:48:48.453904 32577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:39163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=36403
--tserver_master_addrs=127.4.217.62:35027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:48:48.455053 32577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:48:48.456128 32577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:48:48.465988 32577 server_base.cc:1034] running on GCE node
W20260808 04:48:48.465968 32586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:48.465951 32587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:48:48.465945 32593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:48:48.466529 32577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:48:48.467135 32577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:48:48.468331 32577 hybrid_clock.cc:648] HybridClock initialized: now 1786164528468311 us; error 29 us; skew 500 ppm
Aug 08 04:48:48 dist-test-slave-2kcd krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:48:48.471338 32577 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:48:48.472245 32577 webserver.cc:460] Webserver started at http://127.4.217.3:36403/ using document root <none> and password file <none>
I20260808 04:48:48.472867 32577 fs_manager.cc:359] Metadata directory not provided
I20260808 04:48:48.472959 32577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:48:48.476675 32577 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:48:48.478771 32602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:48.480067 32577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:48:48.480165 32577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "897ff2411dc341b691f2c22507a93df6"
format_stamp: "Formatted at 2026-08-08 04:48:06 on dist-test-slave-2kcd"
server_key: "4795a2a72fbe8b1762f59442d5c405cf"
server_key_iv: "c953516038645fac726527f7f8d9c1ef"
server_key_version: "encryptionkey@0"
I20260808 04:48:48.480759 32577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:48:48.492048 32577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:48:48.494860 32577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:48:48.495105 32577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:48:48.495652 32577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:48:48.496512 32577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:48:48.496559 32577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:48.496598 32577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:48:48.496619 32577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:48:48.505699 32577 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:39163
I20260808 04:48:48.505751 305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:39163 every 8 connection(s)
I20260808 04:48:48.506711 32577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:48:48.510587 310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:48.508865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:44131)
0808 04:48:48.509319 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:48.509422 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
W20260808 04:48:48.511025 306 heartbeater.cc:644] Failed to heartbeat to 127.4.217.62:35027 (0 consecutive failures): Network error: Failed to ping master at 127.4.217.62:35027: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
I20260808 04:48:48.511629 310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:48.511406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:56283)
0808 04:48:48.511517 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:48.511566 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 04:48:48.512210 310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:48.511986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:45429)
0808 04:48:48.512092 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:48.512135 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 04:48:48.512432 306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:48:48.513916 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 32577
I20260808 04:48:49.206645 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:49.206217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:38415)
0808 04:48:49.206480 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:49.206533 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
I20260808 04:48:49.331835 313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:49.331404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:49149)
0808 04:48:49.331670 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:49.331729 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260808 04:48:49.513423 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:49.513030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:38649)
0808 04:48:49.513272 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:49.513324 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
I20260808 04:48:49.914695 317 raft_consensus.cc:482] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:48:49.914939 317 raft_consensus.cc:504] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:49.916091 317 leader_election.cc:288] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085), 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163)
Aug 08 04:48:49 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.3@KRBTEST.COM
Aug 08 04:48:49 dist-test-slave-2kcd krb5kdc[30099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.2@KRBTEST.COM
I20260808 04:48:49.929658 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:49.916506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:54763 (local address 127.4.217.2:34085)
0808 04:48:49.917009 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:49.917020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:49.917057 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:48:49.917901 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:49.917909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:49.918255 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:49.918492 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:49.919729 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:49.920681 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:49.921899 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:49.922349 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:49.922534 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:49.925471 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:49.925477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:49.925487 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:49.925515 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:49.927332 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:49.927782 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:49.927788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:49.927813 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:49.927957 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:49.928248 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:49.928252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:49.928253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:49.928674 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:49.928825 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:49.929243 (+ 418us) server_negotiation.cc:299] Negotiation successful
0808 04:48:49.929502 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":84,"threads_started":1}
I20260808 04:48:49.929642 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:49.916298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:34085 (local address 127.4.217.1:54763)
0808 04:48:49.916582 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:49.916601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:49.916715 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:49.918604 (+ 1889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:49.918610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:49.919103 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:48:49.919549 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:49.919564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:49.920859 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:49.920862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:49.921764 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:49.921774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:49.922479 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:49.922482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:49.922678 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:49.923349 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:48:49.923366 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:48:49.925290 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:48:49.927452 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:49.927457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:49.927459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:49.927663 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:49.928059 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:49.928062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:49.928064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:49.928153 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:49.928799 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:48:49.928804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:48:49.928942 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 04:48:49.929124 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 04:48:49.929437 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":104,"spinlock_wait_cycles":34048,"thread_start_us":64,"threads_started":1}
I20260808 04:48:49.929723 323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:49.916541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:59871 (local address 127.4.217.3:39163)
0808 04:48:49.916979 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 04:48:49.917000 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:48:49.917063 (+ 63us) server_negotiation.cc:407] Connection header received
0808 04:48:49.917919 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:48:49.917926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:48:49.918254 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:48:49.918562 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:48:49.919728 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:49.920675 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:49.921895 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:48:49.922346 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:48:49.922531 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:49.925205 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:48:49.925209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:48:49.925221 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:48:49.925264 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:48:49.926931 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:49.927452 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:49.927460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:49.927473 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:49.927579 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:48:49.927950 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:48:49.927953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:48:49.927955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:48:49.928496 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:48:49.928670 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 04:48:49.929253 (+ 583us) server_negotiation.cc:299] Negotiation successful
0808 04:48:49.929531 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":128,"threads_started":1}
I20260808 04:48:49.929757 320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:49.916356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:39163 (local address 127.4.217.1:59871)
0808 04:48:49.916643 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:49.916657 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:48:49.916758 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:48:49.918853 (+ 2095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:48:49.918856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:48:49.919125 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:48:49.919549 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:49.919564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:49.920842 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:49.920846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:49.921764 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:48:49.921774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:48:49.922474 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:48:49.922478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:48:49.922681 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:48:49.923405 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:48:49.923417 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:48:49.924941 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:48:49.927058 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:49.927064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:49.927077 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:49.927326 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:49.927678 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:48:49.927682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:48:49.927684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:48:49.927845 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:48:49.928642 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:48:49.928648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:48:49.928940 (+ 292us) client_negotiation.cc:769] Sending connection context
0808 04:48:49.929124 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 04:48:49.929531 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":312,"spinlock_wait_cycles":24704,"thread_start_us":59,"threads_started":1}
W20260808 04:48:49.932421 32163 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
W20260808 04:48:49.932569 32157 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
I20260808 04:48:49.932678 32157 leader_election.cc:302] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0aabe8e1e94d8ab1062bd0163ca555; no voters: 897ff2411dc341b691f2c22507a93df6, 8ed55f3ed3764245a63367fb6b6651c1
I20260808 04:48:49.932979 317 raft_consensus.cc:2726] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:48:50.207824 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:50.207469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:47551)
0808 04:48:50.207620 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:50.207685 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:48:50.333106 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:50.332611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:43095)
0808 04:48:50.332925 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:50.332997 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":71,"threads_started":1}
I20260808 04:48:50.514477 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:50.514097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:43313)
0808 04:48:50.514332 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:50.514384 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260808 04:48:51.209189 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:51.208707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:55235)
0808 04:48:51.209037 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:51.209093 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":144,"threads_started":1}
I20260808 04:48:51.334213 332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:51.333789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:37949)
0808 04:48:51.334061 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:51.334115 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
I20260808 04:48:51.515707 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:51.515314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:40053)
0808 04:48:51.515560 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:51.515614 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260808 04:48:51.904736 334 raft_consensus.cc:482] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:48:51.904872 334 raft_consensus.cc:504] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:51.905267 334 leader_election.cc:288] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085), 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163)
W20260808 04:48:51.906147 32163 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
W20260808 04:48:51.906255 32157 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
I20260808 04:48:51.906313 32157 leader_election.cc:302] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0aabe8e1e94d8ab1062bd0163ca555; no voters: 897ff2411dc341b691f2c22507a93df6, 8ed55f3ed3764245a63367fb6b6651c1
I20260808 04:48:51.906463 334 raft_consensus.cc:2726] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:48:52.210418 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:52.209965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:38461)
0808 04:48:52.210258 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:52.210315 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":120,"threads_started":1}
I20260808 04:48:52.335393 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:52.334983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:34527)
0808 04:48:52.335233 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:52.335284 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260808 04:48:52.516870 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:52.516382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:44047)
0808 04:48:52.516654 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:52.516707 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
I20260808 04:48:53.211834 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:53.211370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:57613)
0808 04:48:53.211675 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:53.211733 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":87,"threads_started":1}
I20260808 04:48:53.336576 343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:53.336180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:33027)
0808 04:48:53.336426 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:53.336491 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260808 04:48:53.518069 345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:53.517647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:35133)
0808 04:48:53.517910 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:53.517964 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
I20260808 04:48:54.213106 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:54.212601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:49533)
0808 04:48:54.212947 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:54.213007 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":75,"threads_started":1}
I20260808 04:48:54.337716 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:54.337332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:47559)
0808 04:48:54.337577 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:54.337626 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260808 04:48:54.519320 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:54.518905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:55875)
0808 04:48:54.519171 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:54.519223 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260808 04:48:54.797283 349 raft_consensus.cc:482] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:48:54.797418 349 raft_consensus.cc:504] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:54.797825 349 leader_election.cc:288] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085), 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163)
W20260808 04:48:54.798622 32157 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
W20260808 04:48:54.798725 32163 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
I20260808 04:48:54.798833 32163 leader_election.cc:302] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0aabe8e1e94d8ab1062bd0163ca555; no voters: 897ff2411dc341b691f2c22507a93df6, 8ed55f3ed3764245a63367fb6b6651c1
I20260808 04:48:54.799000 349 raft_consensus.cc:2726] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:48:55.214355 350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:55.213887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:38909)
0808 04:48:55.214176 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:55.214231 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260808 04:48:55.338909 351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:55.338498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:54441)
0808 04:48:55.338774 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:55.338823 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":72,"threads_started":1}
I20260808 04:48:55.520445 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:55.520022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:43669)
0808 04:48:55.520295 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:55.520345 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":70,"threads_started":1}
I20260808 04:48:56.215545 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:56.215090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:37449)
0808 04:48:56.215392 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:56.215449 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":78,"threads_started":1}
I20260808 04:48:56.339926 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:56.339559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:41081)
0808 04:48:56.339792 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:56.339842 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260808 04:48:56.521610 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:56.521187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:34505)
0808 04:48:56.521459 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:56.521515 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260808 04:48:57.216771 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:57.216354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:53125)
0808 04:48:57.216613 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:57.216669 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260808 04:48:57.341122 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:57.340669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:48125)
0808 04:48:57.340962 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:57.341036 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":128,"threads_started":1}
I20260808 04:48:57.522804 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:57.522356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:35343)
0808 04:48:57.522636 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:57.522701 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260808 04:48:58.218097 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:58.217694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:46177)
0808 04:48:58.217942 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:58.218002 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260808 04:48:58.342373 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:58.341966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:48389)
0808 04:48:58.342202 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:58.342254 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260808 04:48:58.524183 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:58.523682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:58209)
0808 04:48:58.523988 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:58.524043 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260808 04:48:59.219308 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:59.218879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:41395)
0808 04:48:59.219154 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:59.219209 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
I20260808 04:48:59.343923 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:59.343404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:38101)
0808 04:48:59.343732 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:59.343800 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
I20260808 04:48:59.525544 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:48:59.525052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:60403)
0808 04:48:59.525378 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 04:48:59.525442 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
I20260808 04:48:59.715140 374 raft_consensus.cc:482] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:48:59.715339 374 raft_consensus.cc:504] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:48:59.715821 374 leader_election.cc:288] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085), 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163)
W20260808 04:48:59.716619 32157 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
W20260808 04:48:59.716712 32163 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
I20260808 04:48:59.716857 32163 leader_election.cc:302] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0aabe8e1e94d8ab1062bd0163ca555; no voters: 897ff2411dc341b691f2c22507a93df6, 8ed55f3ed3764245a63367fb6b6651c1
I20260808 04:48:59.717057 374 raft_consensus.cc:2726] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:49:00.220489 375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:00.220075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:43931)
0808 04:49:00.220316 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:00.220381 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260808 04:49:00.345346 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:00.344877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:45149)
0808 04:49:00.345176 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:00.345232 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260808 04:49:00.526765 377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:00.526297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:43553)
0808 04:49:00.526599 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:00.526653 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260808 04:49:01.221638 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:01.221225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:43821)
0808 04:49:01.221477 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:01.221539 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260808 04:49:01.346419 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:01.346037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:57255)
0808 04:49:01.346271 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:01.346323 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260808 04:49:01.528086 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:01.527646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:53237)
0808 04:49:01.527918 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:01.527982 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
I20260808 04:49:01.552439 382 raft_consensus.cc:482] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:49:01.552625 382 raft_consensus.cc:504] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed55f3ed3764245a63367fb6b6651c1" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 34085 } } peers { permanent_uuid: "897ff2411dc341b691f2c22507a93df6" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 39163 } } peers { permanent_uuid: "6b0aabe8e1e94d8ab1062bd0163ca555" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 39389 } }
I20260808 04:49:01.553140 382 leader_election.cc:288] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085), 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163)
W20260808 04:49:01.554010 32163 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 897ff2411dc341b691f2c22507a93df6 (127.4.217.3:39163): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
W20260808 04:49:01.554103 32157 leader_election.cc:341] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ed55f3ed3764245a63367fb6b6651c1 (127.4.217.2:34085): Not found: Tablet not found: c4f8db37cf754dd1bf19a3e612184187
I20260808 04:49:01.554234 32157 leader_election.cc:302] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0aabe8e1e94d8ab1062bd0163ca555; no voters: 897ff2411dc341b691f2c22507a93df6, 8ed55f3ed3764245a63367fb6b6651c1
I20260808 04:49:01.554451 382 raft_consensus.cc:2726] T c4f8db37cf754dd1bf19a3e612184187 P 6b0aabe8e1e94d8ab1062bd0163ca555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:49:02.222910 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:02.222381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:44983)
0808 04:49:02.222704 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:02.222772 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
I20260808 04:49:02.347815 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:02.347356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:44449)
0808 04:49:02.347660 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:02.347717 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
I20260808 04:49:02.529358 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:02.528963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:32947)
0808 04:49:02.529211 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:02.529262 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260808 04:49:03.224351 386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:03.223949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.1:54281)
0808 04:49:03.224197 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:03.224248 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260808 04:49:03.349308 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:03.348907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.2:47325)
0808 04:49:03.349157 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:03.349209 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260808 04:49:03.524523 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 32072
I20260808 04:49:03.531988 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:03.531212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35027 (local address 127.4.217.3:41495)
0808 04:49:03.531762 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:03.531840 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":329,"threads_started":1}
I20260808 04:49:03.542306 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 32359
I20260808 04:49:03.552078 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 32577
2026-08-08T04:49:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59586 ms)
[----------] 30 tests from SecurityITest (247182 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:03 dist-test-slave-2kcd krb5kdc[397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:03 dist-test-slave-2kcd krb5kdc[397](info): set up 2 sockets
Aug 08 04:49:03 dist-test-slave-2kcd krb5kdc[397](info): commencing operation
krb5kdc: starting...
W20260808 04:49:04.821058 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:04 dist-test-slave-2kcd krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:04.945839 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35955
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:05.059156 415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:05.059408 415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:05.059448 415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:05.063433 415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:05.063486 415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:05.063506 415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:05.063563 415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:05.063589 415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:05.068971 415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:05.069947 415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:05.070928 415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:05.079494 426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:05.079509 425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:05.079591 415 server_base.cc:1034] running on GCE node
W20260808 04:49:05.079517 431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:05.080350 415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:05.081439 415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:05.082644 415 hybrid_clock.cc:648] HybridClock initialized: now 1786164545082626 us; error 30 us; skew 500 ppm
Aug 08 04:49:05 dist-test-slave-2kcd krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:05.085417 415 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:05.086226 415 webserver.cc:460] Webserver started at http://127.4.217.62:44559/ using document root <none> and password file <none>
I20260808 04:49:05.086769 415 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:05.086829 415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:05.087067 415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:05.089120 415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "fe68828f68994d1184281cbf39038409"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "76f4e22f7bc462cce9dfec676eec0445"
server_key_iv: "62397c1dc2d957c73aa6c3d7b14b08ff"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.089699 415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "fe68828f68994d1184281cbf39038409"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "76f4e22f7bc462cce9dfec676eec0445"
server_key_iv: "62397c1dc2d957c73aa6c3d7b14b08ff"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.093382 415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:49:05.095713 442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.097007 415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:05.097142 415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "fe68828f68994d1184281cbf39038409"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "76f4e22f7bc462cce9dfec676eec0445"
server_key_iv: "62397c1dc2d957c73aa6c3d7b14b08ff"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.097255 415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:05.119238 415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:05.122447 415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:05.122677 415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:05.129747 415 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:35955
I20260808 04:49:05.129776 518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:35955 every 8 connection(s)
I20260808 04:49:05.130713 415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:05.132448 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 415
I20260808 04:49:05.132608 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:05.133018 4964 external_mini_cluster.cc:1428] Setting key 5cdec80551ee48e6c3f5c64d44c62e6f
I20260808 04:49:05.133864 519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:05.140870 519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409: Bootstrap starting.
Aug 08 04:49:05 dist-test-slave-2kcd krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:05.143298 519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:05.144302 519 log.cc:824] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:05.146704 519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409: No bootstrap required, opened a new log
I20260808 04:49:05.148340 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.134361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.4.217.62:35955)
0808 04:49:05.134789 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:05.134809 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:05.134851 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:49:05.135714 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:05.135738 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:05.136135 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:05.136495 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:05.137254 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.138756 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.139639 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.140311 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.140470 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.142937 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:05.142943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:05.142958 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:05.142989 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:05.145043 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.145566 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.145572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.145578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.145711 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.146052 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.146055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.146056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.146586 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:05.146782 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:05.147136 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 04:49:05.147421 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
I20260808 04:49:05.150924 519 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe68828f68994d1184281cbf39038409" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35955 } }
I20260808 04:49:05.151139 519 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:05.151175 519 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe68828f68994d1184281cbf39038409, State: Initialized, Role: FOLLOWER
I20260808 04:49:05.151574 519 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe68828f68994d1184281cbf39038409" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35955 } }
I20260808 04:49:05.151721 519 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:05.151790 519 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:05.151939 519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:05.152850 519 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe68828f68994d1184281cbf39038409" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35955 } }
I20260808 04:49:05.153205 519 leader_election.cc:302] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe68828f68994d1184281cbf39038409; no voters:
I20260808 04:49:05.153517 519 leader_election.cc:288] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:05.153695 528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:05.154014 528 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [term 1 LEADER]: Becoming Leader. State: Replica: fe68828f68994d1184281cbf39038409, State: Running, Role: LEADER
I20260808 04:49:05.154325 528 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe68828f68994d1184281cbf39038409" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35955 } }
I20260808 04:49:05.154927 519 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:05.155997 532 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe68828f68994d1184281cbf39038409. Latest consensus state: current_term: 1 leader_uuid: "fe68828f68994d1184281cbf39038409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe68828f68994d1184281cbf39038409" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35955 } } }
I20260808 04:49:05.156144 532 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:05.156541 542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:05.156520 531 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe68828f68994d1184281cbf39038409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe68828f68994d1184281cbf39038409" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35955 } } }
I20260808 04:49:05.157351 531 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:05.160084 542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:05.165591 542 catalog_manager.cc:1347] Generated new cluster ID: 2a4bb6d407fe4767913a847783294ccf
I20260808 04:49:05.165668 542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:05.177241 542 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:05.178395 542 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:05.184284 542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409: Generated new TSK 0
I20260808 04:49:05.185045 542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:49:05.215675 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35955
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:05.326396 561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:05.326668 561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:05.326743 561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:05.330696 561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:05.330780 561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:05.330852 561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:49:05.335913 561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:05.336885 561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:05.337854 561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:05.347816 575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:05.347853 577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:05.347908 561 server_base.cc:1034] running on GCE node
W20260808 04:49:05.347828 573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:05.348425 561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:05.349184 561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:05.350376 561 hybrid_clock.cc:648] HybridClock initialized: now 1786164545350355 us; error 33 us; skew 500 ppm
Aug 08 04:49:05 dist-test-slave-2kcd krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:05.353319 561 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:49:05.354223 561 webserver.cc:460] Webserver started at http://127.4.217.1:33539/ using document root <none> and password file <none>
I20260808 04:49:05.354797 561 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:05.354890 561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:05.355134 561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:05.357326 561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "50f4a70f6641484288d84442bfcc4cfb"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "2f21b1148d62d01538ae6c43aa8be9cc"
server_key_iv: "c25669e04a022880a36b538a9ae15b6b"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.357952 561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "50f4a70f6641484288d84442bfcc4cfb"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "2f21b1148d62d01538ae6c43aa8be9cc"
server_key_iv: "c25669e04a022880a36b538a9ae15b6b"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.361667 561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:49:05.364136 587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.365368 561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:05.365511 561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "50f4a70f6641484288d84442bfcc4cfb"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "2f21b1148d62d01538ae6c43aa8be9cc"
server_key_iv: "c25669e04a022880a36b538a9ae15b6b"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.365624 561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:05.378140 561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:05.381441 561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:05.381667 561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:05.382208 561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:05.382994 561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:05.383049 561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.383088 561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:05.383103 561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.392050 561 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:34669
I20260808 04:49:05.392077 769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:34669 every 8 connection(s)
I20260808 04:49:05.392947 561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:49:05.402053 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 561
I20260808 04:49:05.402196 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
Aug 08 04:49:05 dist-test-slave-2kcd krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:05.402565 4964 external_mini_cluster.cc:1428] Setting key 050b9b3ea748fa3f1284466980a1c3e6
I20260808 04:49:05.407155 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.394695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:45779 (local address 127.4.217.62:35955)
0808 04:49:05.394845 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:05.394848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:05.395568 (+ 720us) server_negotiation.cc:407] Connection header received
0808 04:49:05.396954 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:05.396958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:05.397017 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:05.397124 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:05.398233 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.398785 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.399934 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.400180 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.400296 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.403134 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:05.403141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:05.403145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:05.403178 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:05.404705 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.405370 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.405373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.405374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.405458 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.405883 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.405885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.405886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.406051 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:05.406124 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:05.406788 (+ 664us) server_negotiation.cc:299] Negotiation successful
0808 04:49:05.406932 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:49:05.407768 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.394974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35955 (local address 127.4.217.1:45779)
0808 04:49:05.395404 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:05.395446 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:05.396699 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:05.397293 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:05.397299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:05.397633 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:05.398084 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.398098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.398915 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.398918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.399787 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.399796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.400350 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.400354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.400536 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.401140 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:05.401156 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:05.402945 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:05.404850 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:05.404859 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:05.404875 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:05.405251 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:05.405573 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:05.405579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:05.405584 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:05.405780 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:05.406155 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:05.406164 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:05.406511 (+ 347us) client_negotiation.cc:769] Sending connection context
0808 04:49:05.406686 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 04:49:05.406925 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260808 04:49:05.409009 770 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35955
I20260808 04:49:05.409286 770 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:05.409914 770 heartbeater.cc:511] Master 127.4.217.62:35955 requested a full tablet report, sending...
I20260808 04:49:05.411542 463 ts_manager.cc:194] Registered new tserver with Master: 50f4a70f6641484288d84442bfcc4cfb (127.4.217.1:34669)
I20260808 04:49:05.412683 463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:45779
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:49:05.432310 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35955
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:05.548624 779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:05.548925 779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:05.548971 779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:05.552891 779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:05.552943 779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:05.553004 779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:49:05.558470 779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:05.559396 779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:05.560333 779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:05.569769 793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:05.569730 792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:05.569887 779 server_base.cc:1034] running on GCE node
W20260808 04:49:05.569730 795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:05.570382 779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:05.570969 779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:05.572130 779 hybrid_clock.cc:648] HybridClock initialized: now 1786164545572098 us; error 44 us; skew 500 ppm
Aug 08 04:49:05 dist-test-slave-2kcd krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:05.574832 779 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:49:05.575685 779 webserver.cc:460] Webserver started at http://127.4.217.2:44941/ using document root <none> and password file <none>
I20260808 04:49:05.576228 779 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:05.576320 779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:05.576555 779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:05.578523 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "256cc25e428b4b46863d68559a0b8099"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "140c9d9f20ccea756f5d957ac91e6328"
server_key_iv: "1626c6226f56671869f6426970392f7d"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.579073 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "256cc25e428b4b46863d68559a0b8099"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "140c9d9f20ccea756f5d957ac91e6328"
server_key_iv: "1626c6226f56671869f6426970392f7d"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.582628 779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:49:05.584926 805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.586069 779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:05.586203 779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "256cc25e428b4b46863d68559a0b8099"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "140c9d9f20ccea756f5d957ac91e6328"
server_key_iv: "1626c6226f56671869f6426970392f7d"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.586306 779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:05.619966 779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:05.622630 779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:05.622823 779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:05.623368 779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:05.624236 779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:05.624307 779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.624377 779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:05.624433 779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.633323 779 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:43301
I20260808 04:49:05.633352 983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:43301 every 8 connection(s)
I20260808 04:49:05.634155 779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:49:05.639099 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 779
I20260808 04:49:05.639205 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:49:05.639503 4964 external_mini_cluster.cc:1428] Setting key 3e26b7b50ae6c05f4577bf50e3344902
Aug 08 04:49:05 dist-test-slave-2kcd krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:05.648867 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.635925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:49219 (local address 127.4.217.62:35955)
0808 04:49:05.636074 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:05.636078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:05.636937 (+ 859us) server_negotiation.cc:407] Connection header received
0808 04:49:05.638323 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:05.638326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:05.638385 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:05.638465 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:05.639955 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.640571 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.641917 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.642302 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.642422 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.645219 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:05.645222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:05.645224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:05.645248 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:05.646721 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.647297 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.647301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.647302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.647395 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.647745 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.647754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.647757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.647931 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:05.648041 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:05.648534 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 04:49:05.648664 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:49:05.649611 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.636314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35955 (local address 127.4.217.2:49219)
0808 04:49:05.636749 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:05.636828 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:05.638086 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:05.638698 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:05.638705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:05.639128 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:05.639743 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.639761 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.640719 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.640723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.641761 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.641772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.642449 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.642451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.642615 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.643238 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:05.643265 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:05.645048 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:05.646850 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:05.646856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:05.646870 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:05.647175 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:05.647489 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:05.647493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:05.647495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:05.647633 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:05.648041 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:05.648049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:05.648285 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 04:49:05.648472 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 04:49:05.648821 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:49:05.650831 984 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35955
I20260808 04:49:05.651036 984 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:05.651507 984 heartbeater.cc:511] Master 127.4.217.62:35955 requested a full tablet report, sending...
I20260808 04:49:05.652490 463 ts_manager.cc:194] Registered new tserver with Master: 256cc25e428b4b46863d68559a0b8099 (127.4.217.2:43301)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
I20260808 04:49:05.653267 463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:49219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:49:05.668092 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35955
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:05.780464 993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:05.780715 993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:05.780757 993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:05.784840 993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:05.784893 993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:05.784966 993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:49:05.790176 993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:35955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:05.791167 993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:05.792188 993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:05.801802 1006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:05.801973 993 server_base.cc:1034] running on GCE node
W20260808 04:49:05.801826 1002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:05.801802 1001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:05.802482 993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:05.803104 993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:05.804311 993 hybrid_clock.cc:648] HybridClock initialized: now 1786164545804277 us; error 25 us; skew 500 ppm
Aug 08 04:49:05 dist-test-slave-2kcd krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:05.807133 993 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:49:05.808008 993 webserver.cc:460] Webserver started at http://127.4.217.3:33825/ using document root <none> and password file <none>
I20260808 04:49:05.808591 993 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:05.808686 993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:05.808950 993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:05.810928 993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "d7d27432b8584408822fc6863b59e2b7"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "972a73b949770c7c9f26a4c6496ed0ef"
server_key_iv: "d5e71af6543e4190e2b5a15919ffee4f"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.811545 993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "d7d27432b8584408822fc6863b59e2b7"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "972a73b949770c7c9f26a4c6496ed0ef"
server_key_iv: "d5e71af6543e4190e2b5a15919ffee4f"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.817164 993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 04:49:05.821153 1018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.822769 993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:49:05.822892 993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "d7d27432b8584408822fc6863b59e2b7"
format_stamp: "Formatted at 2026-08-08 04:49:05 on dist-test-slave-2kcd"
server_key: "972a73b949770c7c9f26a4c6496ed0ef"
server_key_iv: "d5e71af6543e4190e2b5a15919ffee4f"
server_key_version: "encryptionkey@0"
I20260808 04:49:05.823043 993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:05.835204 993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:05.837909 993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:05.838104 993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:05.838678 993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:05.839479 993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:05.839531 993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.839613 993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:05.839651 993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:05.848337 993 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:36131
I20260808 04:49:05.848366 1205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:36131 every 8 connection(s)
I20260808 04:49:05.849256 993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:49:05.854652 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 993
I20260808 04:49:05.854800 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:49:05.855114 4964 external_mini_cluster.cc:1428] Setting key bd005993635d2656b50c8eec6344fac5
Aug 08 04:49:05 dist-test-slave-2kcd krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:05.862991 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.851134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:51565 (local address 127.4.217.62:35955)
0808 04:49:05.851271 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:05.851275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:05.852081 (+ 806us) server_negotiation.cc:407] Connection header received
0808 04:49:05.853416 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:05.853419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:05.853468 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:05.853544 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:05.854726 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.855375 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.857078 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.857350 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.857473 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.859954 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:05.859957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:05.859959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:05.859983 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:05.861284 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.861740 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.861743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.861744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.861817 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.862124 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.862126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.862127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.862281 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:05.862354 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:05.862778 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 04:49:05.862872 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:49:05.863804 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.851466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35955 (local address 127.4.217.3:51565)
0808 04:49:05.851918 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:05.851961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:05.853221 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:05.853697 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:05.853703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:05.854053 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:05.854563 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.854587 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.855548 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.855551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.856947 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.856958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.857513 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.857528 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.857669 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.858148 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:05.858164 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:05.859805 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:05.861401 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:05.861406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:05.861416 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:05.861655 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:05.861895 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:05.861898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:05.861899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:05.862033 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:05.862365 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:05.862369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:05.862569 (+ 200us) client_negotiation.cc:769] Sending connection context
0808 04:49:05.862730 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 04:49:05.862952 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260808 04:49:05.864940 1206 heartbeater.cc:347] Connected to a master server at 127.4.217.62:35955
I20260808 04:49:05.865187 1206 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:05.865653 1206 heartbeater.cc:511] Master 127.4.217.62:35955 requested a full tablet report, sending...
I20260808 04:49:05.866652 463 ts_manager.cc:194] Registered new tserver with Master: d7d27432b8584408822fc6863b59e2b7 (127.4.217.3:36131)
I20260808 04:49:05.867165 463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:51565
I20260808 04:49:05.869285 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:49:05.877902 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.871211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.4.217.62:35955)
0808 04:49:05.871355 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:05.871360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:05.871522 (+ 162us) server_negotiation.cc:407] Connection header received
0808 04:49:05.871615 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:05.871617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:05.871654 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:05.871716 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:05.872332 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.872864 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.873678 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.873876 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.873965 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.875258 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:05.875264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:05.875270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:05.875299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:05.876475 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.876902 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.876906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.876907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.876980 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:05.877218 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:05.877220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:05.877222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:05.877419 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:05.877518 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:05.877710 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 04:49:05.877798 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:49:05.880610 463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:49:05.883590 463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:49:05.897441 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.891623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:34669 (local address 127.0.0.1:48218)
0808 04:49:05.892108 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:05.892151 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:05.892279 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:05.893391 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:05.893397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:05.893421 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:05.893533 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.893539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.894680 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.894684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.895790 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.895800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.896892 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.896898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.897041 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.897086 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 04:49:05.897197 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 04:49:05.897301 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":90,"threads_started":1}
I20260808 04:49:05.897509 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.892296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.4.217.1:34669)
0808 04:49:05.892619 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:05.892656 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:05.892682 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:49:05.892764 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:05.892772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:05.893002 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:05.893180 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:05.893652 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.894494 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.895915 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.896732 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.896885 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.897076 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:05.897248 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 04:49:05.897347 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
I20260808 04:49:05.897924 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.891753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:36131 (local address 127.0.0.1:34600)
0808 04:49:05.892767 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:05.892801 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:05.892896 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:05.893701 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:05.893704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:05.893720 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:05.893815 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.893821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.895152 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.895154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.896153 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.896177 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.897510 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.897514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.897619 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.897647 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:49:05.897724 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:49:05.897796 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260808 04:49:05.898452 1227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.891889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.4.217.3:36131)
0808 04:49:05.893305 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:05.893319 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:05.893338 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:49:05.893412 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:05.893417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:05.893577 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:05.893728 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:05.893979 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.895019 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.896304 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.897204 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.897308 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.897441 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:05.898212 (+ 771us) server_negotiation.cc:299] Negotiation successful
0808 04:49:05.898306 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1323,"thread_start_us":154,"threads_started":1}
I20260808 04:49:05.900166 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.892331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:43301 (local address 127.0.0.1:48236)
0808 04:49:05.893863 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:05.893877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:05.893962 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:05.894447 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:05.894450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:05.894470 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:05.894578 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.894584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.897015 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.897018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.898184 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:05.898194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.899765 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.899771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:05.899859 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.899880 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:49:05.899977 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 04:49:05.900046 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":104,"threads_started":1}
I20260808 04:49:05.900292 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:05.892499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.4.217.2:43301)
0808 04:49:05.893134 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:05.893145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:05.894055 (+ 910us) server_negotiation.cc:407] Connection header received
0808 04:49:05.894135 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:05.894140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:05.894301 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:05.894456 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:05.895960 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.896851 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.898303 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:05.899469 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:05.899573 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:05.899697 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:05.900044 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 04:49:05.900147 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":107,"threads_started":1}
I20260808 04:49:05.901044 665 tablet_service.cc:1467] Processing CreateTablet for tablet 42ff9fca57184c0e9740a208605f7936 (DEFAULT_TABLE table=test-table [id=5434a8dc08fe416a9eb4fca584458c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:05.901108 1097 tablet_service.cc:1467] Processing CreateTablet for tablet 42ff9fca57184c0e9740a208605f7936 (DEFAULT_TABLE table=test-table [id=5434a8dc08fe416a9eb4fca584458c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:05.901973 665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ff9fca57184c0e9740a208605f7936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:05.901973 1097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ff9fca57184c0e9740a208605f7936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:05.907459 883 tablet_service.cc:1467] Processing CreateTablet for tablet 42ff9fca57184c0e9740a208605f7936 (DEFAULT_TABLE table=test-table [id=5434a8dc08fe416a9eb4fca584458c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:05.908496 883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ff9fca57184c0e9740a208605f7936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:05.909004 1236 tablet_bootstrap.cc:492] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb: Bootstrap starting.
I20260808 04:49:05.909304 1235 tablet_bootstrap.cc:492] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: Bootstrap starting.
I20260808 04:49:05.911005 1235 tablet_bootstrap.cc:654] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:05.911156 1236 tablet_bootstrap.cc:654] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:05.912039 1236 log.cc:824] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:05.912102 1235 log.cc:824] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:05.914070 1235 tablet_bootstrap.cc:492] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: No bootstrap required, opened a new log
I20260808 04:49:05.914070 1236 tablet_bootstrap.cc:492] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb: No bootstrap required, opened a new log
I20260808 04:49:05.914263 1235 ts_tablet_manager.cc:1397] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:49:05.914263 1236 ts_tablet_manager.cc:1397] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:49:05.915248 1241 tablet_bootstrap.cc:492] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: Bootstrap starting.
I20260808 04:49:05.917517 1241 tablet_bootstrap.cc:654] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:05.918181 1241 log.cc:824] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:05.918243 1235 raft_consensus.cc:348] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } }
I20260808 04:49:05.918533 1235 raft_consensus.cc:374] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:05.918618 1235 raft_consensus.cc:729] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d27432b8584408822fc6863b59e2b7, State: Initialized, Role: FOLLOWER
I20260808 04:49:05.918470 1236 raft_consensus.cc:348] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } }
I20260808 04:49:05.918696 1236 raft_consensus.cc:374] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:05.918769 1236 raft_consensus.cc:729] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f4a70f6641484288d84442bfcc4cfb, State: Initialized, Role: FOLLOWER
I20260808 04:49:05.919107 1235 consensus_queue.cc:260] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } }
I20260808 04:49:05.919128 1236 consensus_queue.cc:260] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } }
I20260808 04:49:05.919804 770 heartbeater.cc:503] Master 127.4.217.62:35955 was elected leader, sending a full tablet report...
I20260808 04:49:05.920091 1235 ts_tablet_manager.cc:1428] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:49:05.920091 1236 ts_tablet_manager.cc:1428] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:49:05.920929 1206 heartbeater.cc:503] Master 127.4.217.62:35955 was elected leader, sending a full tablet report...
I20260808 04:49:05.921397 1241 tablet_bootstrap.cc:492] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: No bootstrap required, opened a new log
I20260808 04:49:05.921609 1241 ts_tablet_manager.cc:1397] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:49:05.924746 1241 raft_consensus.cc:348] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } }
I20260808 04:49:05.925012 1241 raft_consensus.cc:374] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:05.925088 1241 raft_consensus.cc:729] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256cc25e428b4b46863d68559a0b8099, State: Initialized, Role: FOLLOWER
I20260808 04:49:05.925427 1241 consensus_queue.cc:260] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } }
I20260808 04:49:05.926011 984 heartbeater.cc:503] Master 127.4.217.62:35955 was elected leader, sending a full tablet report...
I20260808 04:49:05.926198 1241 ts_tablet_manager.cc:1428] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:49:06.039764 1246 raft_consensus.cc:482] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:49:06.040066 1246 raft_consensus.cc:504] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } }
I20260808 04:49:06.041379 1246 leader_election.cc:288] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50f4a70f6641484288d84442bfcc4cfb (127.4.217.1:34669), d7d27432b8584408822fc6863b59e2b7 (127.4.217.3:36131)
I20260808 04:49:06.045140 1227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.041915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:60497 (local address 127.4.217.3:36131)
0808 04:49:06.042035 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:06.042038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:06.042050 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:49:06.042098 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:06.042100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:06.042138 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:06.042241 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:06.042571 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.043071 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.044104 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.044507 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.044640 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.044705 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:06.044941 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 04:49:06.045012 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:49:06.045140 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.041655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:36131 (local address 127.4.217.2:60497)
0808 04:49:06.041764 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:06.041780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:06.041874 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:06.042261 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:06.042264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:06.042309 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:06.042426 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:06.042431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.043191 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.043196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:06.043964 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:06.043973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.044645 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.044649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:06.044754 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.044832 (+ 78us) client_negotiation.cc:769] Sending connection context
0808 04:49:06.044925 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:49:06.045000 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"spinlock_wait_cycles":2304}
I20260808 04:49:06.045326 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.041789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:44825 (local address 127.4.217.1:34669)
0808 04:49:06.041925 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:06.041930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:06.042259 (+ 329us) server_negotiation.cc:407] Connection header received
0808 04:49:06.042426 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:06.042429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:06.042482 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:06.042564 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:06.042839 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.043354 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.044376 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.044758 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.044906 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.044970 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:06.045144 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 04:49:06.045216 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:49:06.045347 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.041654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:34669 (local address 127.4.217.2:44825)
0808 04:49:06.042156 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:06.042166 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:06.042307 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:06.042577 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:06.042579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:06.042595 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:06.042716 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:06.042721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.043477 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.043482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:06.044250 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:06.044258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.044905 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.044909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:06.045014 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.045041 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:49:06.045128 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:49:06.045197 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":112,"spinlock_wait_cycles":22144,"thread_start_us":174,"threads_started":1}
I20260808 04:49:06.045995 698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ff9fca57184c0e9740a208605f7936" candidate_uuid: "256cc25e428b4b46863d68559a0b8099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50f4a70f6641484288d84442bfcc4cfb" is_pre_election: true
I20260808 04:49:06.046011 1130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ff9fca57184c0e9740a208605f7936" candidate_uuid: "256cc25e428b4b46863d68559a0b8099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d27432b8584408822fc6863b59e2b7" is_pre_election: true
I20260808 04:49:06.046304 698 raft_consensus.cc:2454] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 256cc25e428b4b46863d68559a0b8099 in term 0.
I20260808 04:49:06.046326 1130 raft_consensus.cc:2454] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 256cc25e428b4b46863d68559a0b8099 in term 0.
I20260808 04:49:06.046835 806 leader_election.cc:302] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 256cc25e428b4b46863d68559a0b8099, 50f4a70f6641484288d84442bfcc4cfb, d7d27432b8584408822fc6863b59e2b7; no voters:
I20260808 04:49:06.047117 1246 raft_consensus.cc:2781] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:49:06.047199 1246 raft_consensus.cc:482] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:49:06.047228 1246 raft_consensus.cc:3037] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:06.048353 1246 raft_consensus.cc:504] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } }
I20260808 04:49:06.048740 1246 leader_election.cc:288] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [CANDIDATE]: Term 1 election: Requested vote from peers 50f4a70f6641484288d84442bfcc4cfb (127.4.217.1:34669), d7d27432b8584408822fc6863b59e2b7 (127.4.217.3:36131)
I20260808 04:49:06.049094 698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ff9fca57184c0e9740a208605f7936" candidate_uuid: "256cc25e428b4b46863d68559a0b8099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50f4a70f6641484288d84442bfcc4cfb"
I20260808 04:49:06.049142 1130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ff9fca57184c0e9740a208605f7936" candidate_uuid: "256cc25e428b4b46863d68559a0b8099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d27432b8584408822fc6863b59e2b7"
I20260808 04:49:06.049196 698 raft_consensus.cc:3037] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:06.049286 1130 raft_consensus.cc:3037] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:06.050280 698 raft_consensus.cc:2454] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 256cc25e428b4b46863d68559a0b8099 in term 1.
I20260808 04:49:06.050393 1130 raft_consensus.cc:2454] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 256cc25e428b4b46863d68559a0b8099 in term 1.
I20260808 04:49:06.050590 806 leader_election.cc:302] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256cc25e428b4b46863d68559a0b8099, 50f4a70f6641484288d84442bfcc4cfb; no voters:
I20260808 04:49:06.050772 1246 raft_consensus.cc:2781] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:06.050973 1246 raft_consensus.cc:686] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 1 LEADER]: Becoming Leader. State: Replica: 256cc25e428b4b46863d68559a0b8099, State: Running, Role: LEADER
I20260808 04:49:06.051297 1246 consensus_queue.cc:237] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } }
I20260808 04:49:06.054414 463 catalog_manager.cc:5495] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 reported cstate change: term changed from 0 to 1, leader changed from <none> to 256cc25e428b4b46863d68559a0b8099 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "256cc25e428b4b46863d68559a0b8099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "256cc25e428b4b46863d68559a0b8099" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43301 } health_report { overall_health: HEALTHY } } }
I20260808 04:49:06.085332 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.082026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.4.217.2:43301)
0808 04:49:06.082158 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:06.082162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:06.082190 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:49:06.082268 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:06.082270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:06.082328 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:06.082412 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:06.082616 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.083130 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.083965 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.084176 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.084251 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.084541 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:06.085033 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:06.085123 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:06.085177 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 04:49:06.085224 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:49:06.091579 698 raft_consensus.cc:1261] T 42ff9fca57184c0e9740a208605f7936 P 50f4a70f6641484288d84442bfcc4cfb [term 1 FOLLOWER]: Refusing update from remote peer 256cc25e428b4b46863d68559a0b8099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:06.091585 1130 raft_consensus.cc:1261] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [term 1 FOLLOWER]: Refusing update from remote peer 256cc25e428b4b46863d68559a0b8099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:06.092304 1246 consensus_queue.cc:1035] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50f4a70f6641484288d84442bfcc4cfb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:06.092514 1248 consensus_queue.cc:1035] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d27432b8584408822fc6863b59e2b7" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:49:06.100214 1207 tablet.cc:2369] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:49:06.101187 1253 mvcc.cc:205] Tried to move back new op lower bound from 7316129980783738880 to 7316129980628611072. Current Snapshot: MvccSnapshot[applied={T|T < 7316129980783738880 or (T in {7316129980783738880})}]
I20260808 04:49:06.108891 460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:06.109063 460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:06.111415 460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe68828f68994d1184281cbf39038409: Sending DeleteTablet for 3 replicas of tablet 42ff9fca57184c0e9740a208605f7936
I20260808 04:49:06.112142 665 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ff9fca57184c0e9740a208605f7936 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:06 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:48218
I20260808 04:49:06.112370 1097 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ff9fca57184c0e9740a208605f7936 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:06 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:34600
I20260808 04:49:06.112802 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 561
I20260808 04:49:06.112849 1268 tablet_replica.cc:330] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: stopping tablet replica
I20260808 04:49:06.112867 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ff9fca57184c0e9740a208605f7936 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:06 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:48236
I20260808 04:49:06.120507 1268 raft_consensus.cc:2229] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:06.120899 1268 raft_consensus.cc:2258] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:06.121274 1269 tablet_replica.cc:330] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: stopping tablet replica
I20260808 04:49:06.121601 1269 raft_consensus.cc:2229] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:49:06.122011 1269 raft_consensus.cc:2258] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:06.122737 1268 ts_tablet_manager.cc:1905] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:06.123461 1269 ts_tablet_manager.cc:1905] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:06.125716 1268 ts_tablet_manager.cc:1918] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:06.125881 1268 log.cc:1196] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/42ff9fca57184c0e9740a208605f7936
I20260808 04:49:06.126273 1268 ts_tablet_manager.cc:1939] T 42ff9fca57184c0e9740a208605f7936 P d7d27432b8584408822fc6863b59e2b7: Deleting consensus metadata
I20260808 04:49:06.127018 1269 ts_tablet_manager.cc:1918] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:06.127213 1269 log.cc:1196] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/42ff9fca57184c0e9740a208605f7936
I20260808 04:49:06.127568 446 catalog_manager.cc:4841] TS d7d27432b8584408822fc6863b59e2b7 (127.4.217.3:36131): tablet 42ff9fca57184c0e9740a208605f7936 (table test-table [id=5434a8dc08fe416a9eb4fca584458c74]) successfully deleted
I20260808 04:49:06.127641 1269 ts_tablet_manager.cc:1939] T 42ff9fca57184c0e9740a208605f7936 P 256cc25e428b4b46863d68559a0b8099: Deleting consensus metadata
I20260808 04:49:06.128798 446 catalog_manager.cc:4841] TS 256cc25e428b4b46863d68559a0b8099 (127.4.217.2:43301): tablet 42ff9fca57184c0e9740a208605f7936 (table test-table [id=5434a8dc08fe416a9eb4fca584458c74]) successfully deleted
W20260808 04:49:06.128846 443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:34669 (error 108)
I20260808 04:49:06.129606 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 779
I20260808 04:49:06.131984 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.129599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:34669 (local address 127.0.0.1:48222)
0808 04:49:06.131847 (+ 2248us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:06.131912 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:34669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2166}
W20260808 04:49:06.142958 443 catalog_manager.cc:4584] TS 50f4a70f6641484288d84442bfcc4cfb (127.4.217.1:34669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42ff9fca57184c0e9740a208605f7936: Network error: Client connection negotiation failed: client connection to 127.4.217.1:34669: connect: Connection refused (error 111)
I20260808 04:49:06.148435 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 993
I20260808 04:49:06.167807 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 415
2026-08-08T04:49:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2612 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T04:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:06Z Disabled control of system clock
I20260808 04:49:06.202700 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:34151
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:34151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:49:06.314630 1277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:06.314862 1277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:06.314920 1277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:06.318898 1277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:06.318948 1277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:06.318964 1277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:06.319006 1277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:06.319038 1277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:06.323666 1277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:34151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:06.324585 1277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:06.325886 1277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:06.334007 1286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:06.334023 1291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:06.334092 1277 server_base.cc:1034] running on GCE node
W20260808 04:49:06.334031 1287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:06.335069 1277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:06.336032 1277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:06.337193 1277 hybrid_clock.cc:648] HybridClock initialized: now 1786164546337164 us; error 38 us; skew 500 ppm
I20260808 04:49:06.338992 1277 webserver.cc:460] Webserver started at http://127.4.217.62:38393/ using document root <none> and password file <none>
I20260808 04:49:06.339547 1277 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:06.339609 1277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:06.339862 1277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:06.341871 1277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "4581eb0e78504934b227a9213fe38ffd"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "8dfe3fc5dba612359a8042f99608ff26"
server_key_iv: "734029760c211a6481549f0f5767eb38"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.342428 1277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "4581eb0e78504934b227a9213fe38ffd"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "8dfe3fc5dba612359a8042f99608ff26"
server_key_iv: "734029760c211a6481549f0f5767eb38"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.346027 1277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:49:06.348332 1298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:06.349598 1277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:49:06.349772 1277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "4581eb0e78504934b227a9213fe38ffd"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "8dfe3fc5dba612359a8042f99608ff26"
server_key_iv: "734029760c211a6481549f0f5767eb38"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.349889 1277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:06.361738 1277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:06.362347 1277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:06.362558 1277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:06.370357 1277 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:34151
I20260808 04:49:06.370374 1390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:34151 every 8 connection(s)
I20260808 04:49:06.371310 1277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:06.374902 1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:06.378197 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 1277
I20260808 04:49:06.378360 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:06.378664 4964 external_mini_cluster.cc:1428] Setting key a7d415eff18c381fb0aa68d3bc22d50c
I20260808 04:49:06.380761 1392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd: Bootstrap starting.
I20260808 04:49:06.383111 1392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:06.384053 1392 log.cc:824] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:06.386389 1392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd: No bootstrap required, opened a new log
I20260808 04:49:06.388021 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.379968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.4.217.62:34151)
0808 04:49:06.380563 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:06.380579 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:06.380614 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:49:06.381333 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:06.381350 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:06.381727 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:06.382005 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:06.382341 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.383318 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.384651 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.385272 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.385408 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.386269 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:06.386274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:06.386286 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:06.386739 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:06.386794 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:06.386815 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:06.386908 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:06.387065 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 04:49:06.387359 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
I20260808 04:49:06.390422 1392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581eb0e78504934b227a9213fe38ffd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34151 } }
I20260808 04:49:06.390678 1392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:06.390753 1392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4581eb0e78504934b227a9213fe38ffd, State: Initialized, Role: FOLLOWER
I20260808 04:49:06.391166 1392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581eb0e78504934b227a9213fe38ffd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34151 } }
I20260808 04:49:06.391284 1392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:06.391369 1392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:06.391476 1392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:06.392359 1392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581eb0e78504934b227a9213fe38ffd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34151 } }
I20260808 04:49:06.392735 1392 leader_election.cc:302] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4581eb0e78504934b227a9213fe38ffd; no voters:
I20260808 04:49:06.393092 1392 leader_election.cc:288] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:06.393253 1398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:06.393505 1398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [term 1 LEADER]: Becoming Leader. State: Replica: 4581eb0e78504934b227a9213fe38ffd, State: Running, Role: LEADER
I20260808 04:49:06.393959 1398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581eb0e78504934b227a9213fe38ffd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34151 } }
I20260808 04:49:06.394299 1392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:06.395650 1400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4581eb0e78504934b227a9213fe38ffd. Latest consensus state: current_term: 1 leader_uuid: "4581eb0e78504934b227a9213fe38ffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581eb0e78504934b227a9213fe38ffd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34151 } } }
I20260808 04:49:06.395699 1399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4581eb0e78504934b227a9213fe38ffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581eb0e78504934b227a9213fe38ffd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 34151 } } }
I20260808 04:49:06.395776 1400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:06.395848 1399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:06.396600 1412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:06.399712 1412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:06.404758 1412 catalog_manager.cc:1347] Generated new cluster ID: 050208ee3f71405f8efded004030e90e
I20260808 04:49:06.404852 1412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:06.423480 1412 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:06.424753 1412 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:06.437381 1412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd: Generated new TSK 0
I20260808 04:49:06.438254 1412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:06.451265 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34151
--builtin_ntp_servers=127.4.217.20:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:49:06.567324 1423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:06.567619 1423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:06.567687 1423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:06.571787 1423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:06.571859 1423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:06.571949 1423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:49:06.576555 1423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:06.577610 1423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:06.579030 1423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:06.588246 1432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:06.588258 1428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:06.588258 1429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:06.588649 1423 server_base.cc:1034] running on GCE node
I20260808 04:49:06.589046 1423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:06.589648 1423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:06.590830 1423 hybrid_clock.cc:648] HybridClock initialized: now 1786164546590785 us; error 59 us; skew 500 ppm
I20260808 04:49:06.592497 1423 webserver.cc:460] Webserver started at http://127.4.217.1:42439/ using document root <none> and password file <none>
I20260808 04:49:06.593125 1423 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:06.593214 1423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:06.593439 1423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:06.595461 1423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "70605271a388454980652e261ff16474"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "f2c64f085b0a642e6fe9b823024241b6"
server_key_iv: "ef791647deffd4b7b82f359884614ca9"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.596164 1423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "70605271a388454980652e261ff16474"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "f2c64f085b0a642e6fe9b823024241b6"
server_key_iv: "ef791647deffd4b7b82f359884614ca9"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.599809 1423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 04:49:06.602219 1444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:06.603307 1423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:49:06.603437 1423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "70605271a388454980652e261ff16474"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "f2c64f085b0a642e6fe9b823024241b6"
server_key_iv: "ef791647deffd4b7b82f359884614ca9"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.603545 1423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:06.619798 1423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:06.620380 1423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:06.620572 1423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:06.621204 1423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:06.622181 1423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:06.622260 1423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:06.622321 1423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:06.622352 1423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:06.631973 1423 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:36399
I20260808 04:49:06.632033 1644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:36399 every 8 connection(s)
I20260808 04:49:06.633080 1423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:49:06.641935 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.634895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:33969 (local address 127.4.217.62:34151)
0808 04:49:06.635091 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:06.635096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:06.636066 (+ 970us) server_negotiation.cc:407] Connection header received
0808 04:49:06.637471 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:06.637473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:06.637521 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:06.637596 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:06.638475 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.638998 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.640074 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.640291 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.640365 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.641481 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:06.641485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:06.641488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:06.641501 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:06.641520 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:06.641530 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:06.641603 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:06.641740 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 04:49:06.641791 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:49:06.642728 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.635264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34151 (local address 127.4.217.1:33969)
0808 04:49:06.635898 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:06.635961 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:06.637280 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:06.637753 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:06.637760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:06.637843 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:06.638344 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:06.638357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.639113 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.639116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:06.639970 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:06.639978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.640425 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.640428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:06.640569 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.641243 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:49:06.641245 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:06.641283 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:49:06.641342 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:06.641613 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:06.641620 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:06.641624 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:49:06.641715 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 04:49:06.641938 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260808 04:49:06.643272 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 1423
I20260808 04:49:06.643411 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:49:06.643747 4964 external_mini_cluster.cc:1428] Setting key d8ec652271204e0445c3920928686b9c
I20260808 04:49:06.644017 1645 heartbeater.cc:347] Connected to a master server at 127.4.217.62:34151
I20260808 04:49:06.644243 1645 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:06.644767 1645 heartbeater.cc:511] Master 127.4.217.62:34151 requested a full tablet report, sending...
I20260808 04:49:06.645788 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34151
--builtin_ntp_servers=127.4.217.20:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:49:06.646394 1326 ts_manager.cc:194] Registered new tserver with Master: 70605271a388454980652e261ff16474 (127.4.217.1:36399)
I20260808 04:49:06.647611 1326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.1:33969
W20260808 04:49:06.755553 1648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:06.755770 1648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:06.755806 1648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:06.759616 1648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:06.759660 1648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:06.759711 1648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:49:06.764430 1648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:06.765383 1648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:06.766619 1648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:06.775847 1658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:06.775918 1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:06.775978 1655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:06.776151 1648 server_base.cc:1034] running on GCE node
I20260808 04:49:06.776542 1648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:06.777242 1648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:06.778426 1648 hybrid_clock.cc:648] HybridClock initialized: now 1786164546778414 us; error 27 us; skew 500 ppm
I20260808 04:49:06.780151 1648 webserver.cc:460] Webserver started at http://127.4.217.2:39563/ using document root <none> and password file <none>
I20260808 04:49:06.780715 1648 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:06.780833 1648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:06.781083 1648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:06.783041 1648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "752c6e88999c4e1a90452a4055301a68"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "421bf5b1008b32ce1ab360dc84d4d78b"
server_key_iv: "cafb0aaf5ab9456272455dbe7d7989cd"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.783584 1648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "752c6e88999c4e1a90452a4055301a68"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "421bf5b1008b32ce1ab360dc84d4d78b"
server_key_iv: "cafb0aaf5ab9456272455dbe7d7989cd"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.787166 1648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:49:06.789618 1675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:06.790724 1648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:06.790855 1648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "752c6e88999c4e1a90452a4055301a68"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "421bf5b1008b32ce1ab360dc84d4d78b"
server_key_iv: "cafb0aaf5ab9456272455dbe7d7989cd"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.790964 1648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:06.819113 1648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:06.819859 1648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:06.820081 1648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:06.820714 1648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:06.821731 1648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:06.821820 1648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:06.821893 1648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:06.821944 1648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:06.830991 1648 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:43679
I20260808 04:49:06.831030 1866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:43679 every 8 connection(s)
I20260808 04:49:06.831878 1648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:49:06.833021 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 1648
I20260808 04:49:06.833140 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:49:06.833475 4964 external_mini_cluster.cc:1428] Setting key 6831df9b2aa118e430994af6aefefda1
I20260808 04:49:06.835793 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34151
--builtin_ntp_servers=127.4.217.20:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:49:06.842649 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.833867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:39911 (local address 127.4.217.62:34151)
0808 04:49:06.834020 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:06.834024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:06.835288 (+ 1264us) server_negotiation.cc:407] Connection header received
0808 04:49:06.837340 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:06.837343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:06.837387 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:06.837447 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:06.838460 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.839290 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.840399 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.840833 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.840963 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.842050 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:06.842057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:06.842062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:06.842081 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:06.842108 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:06.842121 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:06.842212 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:06.842362 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 04:49:06.842439 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:49:06.843381 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:06.834347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34151 (local address 127.4.217.2:39911)
0808 04:49:06.835149 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:06.835202 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:06.837122 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:06.837645 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:06.837656 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:06.837773 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:06.838315 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:06.838328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.839430 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.839432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:06.840271 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:06.840280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:06.841000 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:06.841003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:06.841141 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:06.841780 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:49:06.841782 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:06.841819 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:49:06.841881 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:06.842216 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:06.842223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:06.842228 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:49:06.842321 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:49:06.842544 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":152,"threads_started":1}
I20260808 04:49:06.844539 1867 heartbeater.cc:347] Connected to a master server at 127.4.217.62:34151
I20260808 04:49:06.844766 1867 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:06.845329 1867 heartbeater.cc:511] Master 127.4.217.62:34151 requested a full tablet report, sending...
I20260808 04:49:06.846396 1326 ts_manager.cc:194] Registered new tserver with Master: 752c6e88999c4e1a90452a4055301a68 (127.4.217.2:43679)
I20260808 04:49:06.846984 1326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.2:39911
W20260808 04:49:06.950825 1871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:06.951057 1871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:06.951094 1871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:06.955272 1871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:06.955320 1871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:06.955420 1871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:49:06.960044 1871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:06.961125 1871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:06.962410 1871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:06.972252 1881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:06.972317 1871 server_base.cc:1034] running on GCE node
W20260808 04:49:06.972252 1877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:06.972501 1878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:06.973037 1871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:06.973629 1871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:06.974815 1871 hybrid_clock.cc:648] HybridClock initialized: now 1786164546974800 us; error 36 us; skew 500 ppm
I20260808 04:49:06.976553 1871 webserver.cc:460] Webserver started at http://127.4.217.3:39899/ using document root <none> and password file <none>
I20260808 04:49:06.977178 1871 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:06.977270 1871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:06.977485 1871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:06.979425 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "85ac14f563844b5c88509829b6a86195"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "3f7bbb03211d2480b6672ce514f8a043"
server_key_iv: "7fa955ca27b68c0488d193fe687cd18f"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.979972 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "85ac14f563844b5c88509829b6a86195"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "3f7bbb03211d2480b6672ce514f8a043"
server_key_iv: "7fa955ca27b68c0488d193fe687cd18f"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.983506 1871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 04:49:06.985833 1893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:06.986979 1871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:49:06.987112 1871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "85ac14f563844b5c88509829b6a86195"
format_stamp: "Formatted at 2026-08-08 04:49:06 on dist-test-slave-2kcd"
server_key: "3f7bbb03211d2480b6672ce514f8a043"
server_key_iv: "7fa955ca27b68c0488d193fe687cd18f"
server_key_version: "encryptionkey@0"
I20260808 04:49:06.987227 1871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:07.005710 1871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:07.006376 1871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:07.006601 1871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:07.007192 1871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:07.008193 1871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:07.008266 1871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:07.008342 1871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:07.008385 1871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:07.017342 1871 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:40351
I20260808 04:49:07.017367 2062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:40351 every 8 connection(s)
I20260808 04:49:07.018249 1871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:49:07.023320 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 1871
I20260808 04:49:07.023456 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:49:07.023754 4964 external_mini_cluster.cc:1428] Setting key 155191290b370eaa9c4d06cf3ed28a69
I20260808 04:49:07.028051 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.020097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:55337 (local address 127.4.217.62:34151)
0808 04:49:07.020266 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:07.020271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:07.021329 (+ 1058us) server_negotiation.cc:407] Connection header received
0808 04:49:07.022758 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:07.022760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:07.022808 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:07.022887 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:07.024054 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.024822 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.026206 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.026443 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.026524 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.027575 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:07.027578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:07.027580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:07.027591 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:07.027609 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:07.027617 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:07.027676 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:07.027880 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 04:49:07.027937 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:49:07.029022 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.020442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:34151 (local address 127.4.217.3:55337)
0808 04:49:07.021175 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:07.021216 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:07.022554 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:07.023086 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:07.023094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:07.023200 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:07.023897 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.023918 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.024963 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.024967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.026091 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.026101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.026597 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.026600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.026740 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.027328 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:49:07.027330 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:07.027372 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:49:07.027434 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:07.027718 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:07.027727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:07.027732 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:49:07.027863 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 04:49:07.028109 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260808 04:49:07.030074 2063 heartbeater.cc:347] Connected to a master server at 127.4.217.62:34151
I20260808 04:49:07.030310 2063 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:07.030805 2063 heartbeater.cc:511] Master 127.4.217.62:34151 requested a full tablet report, sending...
I20260808 04:49:07.031782 1326 ts_manager.cc:194] Registered new tserver with Master: 85ac14f563844b5c88509829b6a86195 (127.4.217.3:40351)
I20260808 04:49:07.032339 1326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.3:55337
I20260808 04:49:07.037705 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:49:07.043423 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.039699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.4.217.62:34151)
0808 04:49:07.039844 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:07.039849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:07.039991 (+ 142us) server_negotiation.cc:407] Connection header received
0808 04:49:07.040073 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:07.040076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:07.040113 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:07.040179 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:07.040435 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.040986 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.041872 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.042111 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.042211 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.043028 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:07.043031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:07.043033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:07.043044 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:07.043060 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:07.043068 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:07.043127 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:07.043208 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 04:49:07.043253 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:49:07.046430 1326 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:49:07.049582 1326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:49:07.064054 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.059299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.4.217.3:40351)
0808 04:49:07.059911 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:07.059926 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:07.059951 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:49:07.060034 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:07.060039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:07.060206 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:07.060361 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:07.060594 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.061598 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.062871 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.063384 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.063515 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.063704 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:07.063797 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 04:49:07.063888 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":169,"threads_started":1}
I20260808 04:49:07.064039 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.058747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:40351 (local address 127.0.0.1:41478)
0808 04:49:07.059575 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:07.059588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:07.059662 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:07.060362 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:07.060365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:07.060385 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:07.060480 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.060485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.061705 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.061710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.062734 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.062744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.063522 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.063525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.063628 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.063665 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:49:07.063772 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 04:49:07.063886 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":100,"threads_started":1}
I20260808 04:49:07.064548 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.058020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:36399 (local address 127.0.0.1:34880)
0808 04:49:07.059112 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:07.059131 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:07.059245 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:07.059846 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:07.059849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:07.059871 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:07.059949 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.059955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.061345 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.061349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.062689 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.062694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.063966 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.063970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.064311 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.064332 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:49:07.064391 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 04:49:07.064440 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":128,"thread_start_us":96,"threads_started":1}
I20260808 04:49:07.064548 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.057896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:43679 (local address 127.0.0.1:51946)
0808 04:49:07.058539 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:07.058588 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:07.058729 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:07.059533 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:07.059540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:07.059563 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:07.059707 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.059716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.060943 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.060946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.062680 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.062693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.063966 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.063970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.064159 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.064231 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 04:49:07.064306 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:49:07.064385 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":137,"thread_start_us":100,"threads_started":1}
I20260808 04:49:07.065080 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.058723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.4.217.1:36399)
0808 04:49:07.059028 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:07.059040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:07.059347 (+ 307us) server_negotiation.cc:407] Connection header received
0808 04:49:07.059436 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:07.059442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:07.059627 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:07.059775 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:07.060066 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.061242 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.062932 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.063644 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.063732 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.063895 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:07.064815 (+ 920us) server_negotiation.cc:299] Negotiation successful
0808 04:49:07.064924 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
I20260808 04:49:07.064915 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.058032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.4.217.2:43679)
0808 04:49:07.058397 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:07.058418 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:07.058880 (+ 462us) server_negotiation.cc:407] Connection header received
0808 04:49:07.058970 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:07.058975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:07.059180 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:07.059333 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:07.059816 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.060834 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.062817 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.063607 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.063722 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.063894 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:07.064618 (+ 724us) server_negotiation.cc:299] Negotiation successful
0808 04:49:07.064731 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":142,"threads_started":1}
I20260808 04:49:07.067246 1973 tablet_service.cc:1467] Processing CreateTablet for tablet 26e7805ecfb347388c92f1fd32845d65 (DEFAULT_TABLE table=test-table [id=eb5038487e85412093737675ab5936f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:07.067607 1528 tablet_service.cc:1467] Processing CreateTablet for tablet 26e7805ecfb347388c92f1fd32845d65 (DEFAULT_TABLE table=test-table [id=eb5038487e85412093737675ab5936f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:07.067607 1769 tablet_service.cc:1467] Processing CreateTablet for tablet 26e7805ecfb347388c92f1fd32845d65 (DEFAULT_TABLE table=test-table [id=eb5038487e85412093737675ab5936f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:07.068262 1973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e7805ecfb347388c92f1fd32845d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:07.068593 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e7805ecfb347388c92f1fd32845d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:07.068593 1769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e7805ecfb347388c92f1fd32845d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:07.075472 2095 tablet_bootstrap.cc:492] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474: Bootstrap starting.
I20260808 04:49:07.076165 2094 tablet_bootstrap.cc:492] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: Bootstrap starting.
I20260808 04:49:07.077124 2098 tablet_bootstrap.cc:492] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: Bootstrap starting.
I20260808 04:49:07.077845 2095 tablet_bootstrap.cc:654] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:07.078269 2094 tablet_bootstrap.cc:654] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:07.078758 2095 log.cc:824] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:07.078830 2098 tablet_bootstrap.cc:654] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:07.079070 2094 log.cc:824] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:07.079641 2098 log.cc:824] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:07.080967 2094 tablet_bootstrap.cc:492] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: No bootstrap required, opened a new log
I20260808 04:49:07.080967 2095 tablet_bootstrap.cc:492] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474: No bootstrap required, opened a new log
I20260808 04:49:07.081163 2095 ts_tablet_manager.cc:1397] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:49:07.081163 2094 ts_tablet_manager.cc:1397] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:49:07.081437 2098 tablet_bootstrap.cc:492] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: No bootstrap required, opened a new log
I20260808 04:49:07.081615 2098 ts_tablet_manager.cc:1397] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:49:07.084648 2098 raft_consensus.cc:348] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } }
I20260808 04:49:07.084937 2098 raft_consensus.cc:374] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:07.084993 2098 raft_consensus.cc:729] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752c6e88999c4e1a90452a4055301a68, State: Initialized, Role: FOLLOWER
I20260808 04:49:07.085462 2098 consensus_queue.cc:260] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } }
I20260808 04:49:07.085438 2095 raft_consensus.cc:348] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } }
I20260808 04:49:07.085438 2094 raft_consensus.cc:348] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } }
I20260808 04:49:07.085683 2094 raft_consensus.cc:374] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:07.085683 2095 raft_consensus.cc:374] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:07.085740 2094 raft_consensus.cc:729] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ac14f563844b5c88509829b6a86195, State: Initialized, Role: FOLLOWER
I20260808 04:49:07.085748 2095 raft_consensus.cc:729] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70605271a388454980652e261ff16474, State: Initialized, Role: FOLLOWER
I20260808 04:49:07.086151 1867 heartbeater.cc:503] Master 127.4.217.62:34151 was elected leader, sending a full tablet report...
I20260808 04:49:07.086195 2095 consensus_queue.cc:260] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } }
I20260808 04:49:07.086195 2094 consensus_queue.cc:260] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } }
I20260808 04:49:07.086490 2098 ts_tablet_manager.cc:1428] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:49:07.087081 2095 ts_tablet_manager.cc:1428] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:49:07.087081 2094 ts_tablet_manager.cc:1428] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260808 04:49:07.087402 1645 heartbeater.cc:503] Master 127.4.217.62:34151 was elected leader, sending a full tablet report...
I20260808 04:49:07.087456 2063 heartbeater.cc:503] Master 127.4.217.62:34151 was elected leader, sending a full tablet report...
I20260808 04:49:07.102344 2108 raft_consensus.cc:482] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:49:07.102531 2108 raft_consensus.cc:504] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } }
I20260808 04:49:07.103605 2108 leader_election.cc:288] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 752c6e88999c4e1a90452a4055301a68 (127.4.217.2:43679), 85ac14f563844b5c88509829b6a86195 (127.4.217.3:40351)
I20260808 04:49:07.107707 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.103912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:42663 (local address 127.4.217.2:43679)
0808 04:49:07.104040 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:07.104044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:07.104060 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:49:07.104212 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:07.104215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:07.104277 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:07.104358 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:07.104718 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.105366 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.106663 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.107118 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.107250 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.107310 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:07.107518 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 04:49:07.107592 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:49:07.107707 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.103830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:43679 (local address 127.4.217.1:42663)
0808 04:49:07.103969 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:07.103987 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:07.104081 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:07.104414 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:07.104418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:07.104440 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:07.104585 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.104593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.105494 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.105498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.106532 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.106542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.107261 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.107265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.107373 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.107396 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:49:07.107496 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:49:07.107574 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":3}
I20260808 04:49:07.107846 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.103948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:46273 (local address 127.4.217.3:40351)
0808 04:49:07.104067 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:07.104073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:07.104384 (+ 311us) server_negotiation.cc:407] Connection header received
0808 04:49:07.104447 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:07.104450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:07.104493 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:07.104580 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:07.104959 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.105511 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.106691 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.107150 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.107263 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.107340 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:07.107620 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 04:49:07.107689 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:49:07.108256 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.103830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:40351 (local address 127.4.217.1:46273)
0808 04:49:07.104226 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:07.104239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:07.104345 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:07.104599 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:07.104602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:07.104618 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:07.104769 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.104803 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.105623 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.105626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.106540 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:07.106550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.107404 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.107408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:07.107494 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.107512 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:49:07.107600 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 04:49:07.107667 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":222,"thread_start_us":103,"threads_started":1}
I20260808 04:49:07.108479 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e7805ecfb347388c92f1fd32845d65" candidate_uuid: "70605271a388454980652e261ff16474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752c6e88999c4e1a90452a4055301a68" is_pre_election: true
I20260808 04:49:07.108829 1802 raft_consensus.cc:2454] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70605271a388454980652e261ff16474 in term 0.
I20260808 04:49:07.108928 2000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e7805ecfb347388c92f1fd32845d65" candidate_uuid: "70605271a388454980652e261ff16474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ac14f563844b5c88509829b6a86195" is_pre_election: true
I20260808 04:49:07.109231 2000 raft_consensus.cc:2454] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70605271a388454980652e261ff16474 in term 0.
I20260808 04:49:07.109385 1451 leader_election.cc:302] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70605271a388454980652e261ff16474, 752c6e88999c4e1a90452a4055301a68; no voters:
I20260808 04:49:07.109659 2108 raft_consensus.cc:2781] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:49:07.109746 2108 raft_consensus.cc:482] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:49:07.109790 2108 raft_consensus.cc:3037] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:07.110978 2108 raft_consensus.cc:504] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } }
I20260808 04:49:07.111315 2108 leader_election.cc:288] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [CANDIDATE]: Term 1 election: Requested vote from peers 752c6e88999c4e1a90452a4055301a68 (127.4.217.2:43679), 85ac14f563844b5c88509829b6a86195 (127.4.217.3:40351)
I20260808 04:49:07.111629 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e7805ecfb347388c92f1fd32845d65" candidate_uuid: "70605271a388454980652e261ff16474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752c6e88999c4e1a90452a4055301a68"
I20260808 04:49:07.111776 1802 raft_consensus.cc:3037] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:07.111754 2000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e7805ecfb347388c92f1fd32845d65" candidate_uuid: "70605271a388454980652e261ff16474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ac14f563844b5c88509829b6a86195"
I20260808 04:49:07.111938 2000 raft_consensus.cc:3037] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:07.112890 1802 raft_consensus.cc:2454] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70605271a388454980652e261ff16474 in term 1.
I20260808 04:49:07.113116 2000 raft_consensus.cc:2454] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70605271a388454980652e261ff16474 in term 1.
I20260808 04:49:07.113199 1451 leader_election.cc:302] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70605271a388454980652e261ff16474, 752c6e88999c4e1a90452a4055301a68; no voters:
I20260808 04:49:07.113514 2108 raft_consensus.cc:2781] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:07.113739 2108 raft_consensus.cc:686] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [term 1 LEADER]: Becoming Leader. State: Replica: 70605271a388454980652e261ff16474, State: Running, Role: LEADER
I20260808 04:49:07.114080 2108 consensus_queue.cc:237] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } }
I20260808 04:49:07.117261 1321 catalog_manager.cc:5495] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70605271a388454980652e261ff16474 (127.4.217.1). New cstate: current_term: 1 leader_uuid: "70605271a388454980652e261ff16474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70605271a388454980652e261ff16474" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 36399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 } health_report { overall_health: UNKNOWN } } }
I20260808 04:49:07.130870 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.127529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.4.217.1:36399)
0808 04:49:07.127658 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:07.127661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:07.127674 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:49:07.127818 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:07.127820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:07.127871 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:07.127958 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:07.128238 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.128729 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.129596 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:07.129796 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:07.129870 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:07.130190 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:07.130564 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:07.130652 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:07.130712 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 04:49:07.130753 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 04:49:07.134210 1646 tablet.cc:2369] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:49:07.136708 1802 raft_consensus.cc:1261] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [term 1 FOLLOWER]: Refusing update from remote peer 70605271a388454980652e261ff16474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:07.136718 2000 raft_consensus.cc:1261] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [term 1 FOLLOWER]: Refusing update from remote peer 70605271a388454980652e261ff16474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:07.137444 2108 consensus_queue.cc:1035] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "752c6e88999c4e1a90452a4055301a68" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:07.137648 2114 consensus_queue.cc:1035] T 26e7805ecfb347388c92f1fd32845d65 P 70605271a388454980652e261ff16474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ac14f563844b5c88509829b6a86195" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 40351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:07.145090 2120 mvcc.cc:205] Tried to move back new op lower bound from 7316129985064972288 to 7316129984981999616. Current Snapshot: MvccSnapshot[applied={T|T < 7316129985064972288}]
I20260808 04:49:07.146081 2122 mvcc.cc:205] Tried to move back new op lower bound from 7316129985064972288 to 7316129984981999616. Current Snapshot: MvccSnapshot[applied={T|T < 7316129985064972288}]
I20260808 04:49:07.155414 1320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:07.155627 1320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:07.158397 1320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4581eb0e78504934b227a9213fe38ffd: Sending DeleteTablet for 3 replicas of tablet 26e7805ecfb347388c92f1fd32845d65
I20260808 04:49:07.159210 1528 tablet_service.cc:1514] Processing DeleteTablet for tablet 26e7805ecfb347388c92f1fd32845d65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:07 UTC) from {username='slave'} at 127.0.0.1:34880
I20260808 04:49:07.159210 1769 tablet_service.cc:1514] Processing DeleteTablet for tablet 26e7805ecfb347388c92f1fd32845d65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:07 UTC) from {username='slave'} at 127.0.0.1:51946
I20260808 04:49:07.159525 1973 tablet_service.cc:1514] Processing DeleteTablet for tablet 26e7805ecfb347388c92f1fd32845d65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:07 UTC) from {username='slave'} at 127.0.0.1:41478
I20260808 04:49:07.159926 2145 tablet_replica.cc:330] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: stopping tablet replica
I20260808 04:49:07.160032 2146 tablet_replica.cc:330] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: stopping tablet replica
I20260808 04:49:07.160316 2145 raft_consensus.cc:2229] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:07.160375 2146 raft_consensus.cc:2229] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:07.160486 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 1423
I20260808 04:49:07.160717 2146 raft_consensus.cc:2258] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:07.163919 2145 raft_consensus.cc:2258] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:07.167457 2145 ts_tablet_manager.cc:1905] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:07.168051 2146 ts_tablet_manager.cc:1905] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:07.169925 2145 ts_tablet_manager.cc:1918] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:07.170045 2145 log.cc:1196] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/26e7805ecfb347388c92f1fd32845d65
I20260808 04:49:07.170339 2145 ts_tablet_manager.cc:1939] T 26e7805ecfb347388c92f1fd32845d65 P 752c6e88999c4e1a90452a4055301a68: Deleting consensus metadata
I20260808 04:49:07.171450 1303 catalog_manager.cc:4841] TS 752c6e88999c4e1a90452a4055301a68 (127.4.217.2:43679): tablet 26e7805ecfb347388c92f1fd32845d65 (table test-table [id=eb5038487e85412093737675ab5936f2]) successfully deleted
I20260808 04:49:07.171797 2146 ts_tablet_manager.cc:1918] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:07.171916 2146 log.cc:1196] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/26e7805ecfb347388c92f1fd32845d65
I20260808 04:49:07.172271 2146 ts_tablet_manager.cc:1939] T 26e7805ecfb347388c92f1fd32845d65 P 85ac14f563844b5c88509829b6a86195: Deleting consensus metadata
I20260808 04:49:07.173593 1302 catalog_manager.cc:4841] TS 85ac14f563844b5c88509829b6a86195 (127.4.217.3:40351): tablet 26e7805ecfb347388c92f1fd32845d65 (table test-table [id=eb5038487e85412093737675ab5936f2]) successfully deleted
W20260808 04:49:07.176189 1301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:36399 (error 108)
I20260808 04:49:07.176508 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 1648
I20260808 04:49:07.179898 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:07.179575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:36399 (local address 127.0.0.1:34892)
0808 04:49:07.179717 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:07.179796 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:49:07.181250 1301 catalog_manager.cc:4584] TS 70605271a388454980652e261ff16474 (127.4.217.1:36399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26e7805ecfb347388c92f1fd32845d65: Network error: Client connection negotiation failed: client connection to 127.4.217.1:36399: connect: Connection refused (error 111)
I20260808 04:49:07.187834 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 1871
I20260808 04:49:07.212472 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 1277
2026-08-08T04:49:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1043 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:07 dist-test-slave-2kcd krb5kdc[2155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:07 dist-test-slave-2kcd krb5kdc[2155](info): set up 2 sockets
Aug 08 04:49:07 dist-test-slave-2kcd krb5kdc[2155](info): commencing operation
krb5kdc: starting...
W20260808 04:49:08.472321 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:08 dist-test-slave-2kcd krb5kdc[2155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:08.600013 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:37415
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:08.713979 2174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:08.714278 2174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:08.714363 2174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:08.718889 2174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:08.718971 2174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:08.719010 2174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:08.719054 2174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:08.719089 2174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:08.724155 2174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:37415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:08.725260 2174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:08.726315 2174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:08.734988 2185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:08.735066 2183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:08.735222 2174 server_base.cc:1034] running on GCE node
W20260808 04:49:08.734998 2188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:08.735935 2174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:08.737093 2174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:08.738256 2174 hybrid_clock.cc:648] HybridClock initialized: now 1786164548738245 us; error 25 us; skew 500 ppm
Aug 08 04:49:08 dist-test-slave-2kcd krb5kdc[2155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164548, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:08.746241 2174 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:08.747164 2174 webserver.cc:460] Webserver started at http://127.4.217.62:43559/ using document root <none> and password file <none>
I20260808 04:49:08.747757 2174 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:08.747848 2174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:08.748103 2174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:08.750229 2174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "a3f83332ba3e44668cab0eaeb55a85c5"
format_stamp: "Formatted at 2026-08-08 04:49:08 on dist-test-slave-2kcd"
server_key: "12e310550e9f70f85f68bba02acb0f15"
server_key_iv: "76d2fc7ec9456973d52f28255caedca8"
server_key_version: "encryptionkey@0"
I20260808 04:49:08.750808 2174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "a3f83332ba3e44668cab0eaeb55a85c5"
format_stamp: "Formatted at 2026-08-08 04:49:08 on dist-test-slave-2kcd"
server_key: "12e310550e9f70f85f68bba02acb0f15"
server_key_iv: "76d2fc7ec9456973d52f28255caedca8"
server_key_version: "encryptionkey@0"
I20260808 04:49:08.754492 2174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 04:49:08.756769 2197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:08.757961 2174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:49:08.758117 2174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "a3f83332ba3e44668cab0eaeb55a85c5"
format_stamp: "Formatted at 2026-08-08 04:49:08 on dist-test-slave-2kcd"
server_key: "12e310550e9f70f85f68bba02acb0f15"
server_key_iv: "76d2fc7ec9456973d52f28255caedca8"
server_key_version: "encryptionkey@0"
I20260808 04:49:08.758227 2174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:08.774087 2174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:08.776687 2174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:08.776921 2174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:08.783762 2174 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:37415
I20260808 04:49:08.783797 2288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:37415 every 8 connection(s)
I20260808 04:49:08.784650 2174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:08.787199 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 2174
I20260808 04:49:08.787352 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:08.787493 2289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:08.787678 4964 external_mini_cluster.cc:1428] Setting key 38c93a7f24b55ad27542918a00e1253f
I20260808 04:49:08.793732 2289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5: Bootstrap starting.
I20260808 04:49:08.796182 2289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5: Neither blocks nor log segments found. Creating new log.
Aug 08 04:49:08 dist-test-slave-2kcd krb5kdc[2155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:08.797101 2289 log.cc:824] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:08.799256 2289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5: No bootstrap required, opened a new log
I20260808 04:49:08.801440 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:08.789100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.4.217.62:37415)
0808 04:49:08.789547 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:08.789561 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:08.789594 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:49:08.790214 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:08.790229 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:08.790533 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:08.790845 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:08.791662 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:08.792621 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:08.793532 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:08.794038 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:08.794202 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:08.796911 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:08.796916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:08.796929 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:08.796954 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:08.798640 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:08.799179 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:08.799184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:08.799188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:08.799288 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:08.799603 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:08.799612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:08.799613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:08.799959 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:08.800161 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:08.800454 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 04:49:08.800725 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":151,"threads_started":1}
I20260808 04:49:08.802979 2289 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f83332ba3e44668cab0eaeb55a85c5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37415 } }
I20260808 04:49:08.803232 2289 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:08.803295 2289 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f83332ba3e44668cab0eaeb55a85c5, State: Initialized, Role: FOLLOWER
I20260808 04:49:08.803764 2289 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f83332ba3e44668cab0eaeb55a85c5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37415 } }
I20260808 04:49:08.803892 2289 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:08.803941 2289 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:08.804010 2289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:08.804836 2289 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f83332ba3e44668cab0eaeb55a85c5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37415 } }
I20260808 04:49:08.805202 2289 leader_election.cc:302] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3f83332ba3e44668cab0eaeb55a85c5; no voters:
I20260808 04:49:08.805552 2289 leader_election.cc:288] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:08.805634 2301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:08.805859 2301 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [term 1 LEADER]: Becoming Leader. State: Replica: a3f83332ba3e44668cab0eaeb55a85c5, State: Running, Role: LEADER
I20260808 04:49:08.806177 2301 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f83332ba3e44668cab0eaeb55a85c5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37415 } }
I20260808 04:49:08.806804 2289 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:08.807849 2304 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3f83332ba3e44668cab0eaeb55a85c5. Latest consensus state: current_term: 1 leader_uuid: "a3f83332ba3e44668cab0eaeb55a85c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f83332ba3e44668cab0eaeb55a85c5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37415 } } }
I20260808 04:49:08.808001 2304 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:08.807849 2303 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3f83332ba3e44668cab0eaeb55a85c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f83332ba3e44668cab0eaeb55a85c5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 37415 } } }
I20260808 04:49:08.808113 2303 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:08.808390 2315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:08.811098 2315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:08.815995 2315 catalog_manager.cc:1347] Generated new cluster ID: dcb9b787d33743df9b1a747096cedab1
I20260808 04:49:08.816057 2315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:08.835767 2315 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:08.837018 2315 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:08.842226 2315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5: Generated new TSK 0
I20260808 04:49:08.842934 2315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:49:08.881513 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37415
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:08.994155 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:08.994393 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:08.994436 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:08.998466 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:08.998528 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:08.998615 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:49:09.003827 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:09.004896 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:09.005928 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:49:09.015960 2333 server_base.cc:1034] running on GCE node
W20260808 04:49:09.015874 2341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:09.015874 2345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:09.015883 2342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:09.016563 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:09.017256 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:09.018443 2333 hybrid_clock.cc:648] HybridClock initialized: now 1786164549018431 us; error 27 us; skew 500 ppm
Aug 08 04:49:09 dist-test-slave-2kcd krb5kdc[2155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164549, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:09.021314 2333 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:49:09.022234 2333 webserver.cc:460] Webserver started at http://127.4.217.1:38217/ using document root <none> and password file <none>
I20260808 04:49:09.022820 2333 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:09.022881 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:09.023115 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:09.025216 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "88a28bc1176d4424a0fd3a278bc1a55e"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "f6a00d23fe2c3c3df767deda381fc8ea"
server_key_iv: "b1e432e6583cfc5b2eb4f335c1c824c4"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.025780 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "88a28bc1176d4424a0fd3a278bc1a55e"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "f6a00d23fe2c3c3df767deda381fc8ea"
server_key_iv: "b1e432e6583cfc5b2eb4f335c1c824c4"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.029544 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:49:09.032020 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.033221 2333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:09.033351 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "88a28bc1176d4424a0fd3a278bc1a55e"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "f6a00d23fe2c3c3df767deda381fc8ea"
server_key_iv: "b1e432e6583cfc5b2eb4f335c1c824c4"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.033459 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:09.049824 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:09.053215 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:09.053423 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:09.054040 2333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:09.054888 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:09.054955 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.055025 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:09.055074 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.064405 2333 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:46159
I20260808 04:49:09.064440 2528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:46159 every 8 connection(s)
I20260808 04:49:09.065385 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:49:09.068015 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 2333
I20260808 04:49:09.068176 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:49:09.068478 4964 external_mini_cluster.cc:1428] Setting key dc8a2709d4061617dd4df4f01235e2c0
Aug 08 04:49:09 dist-test-slave-2kcd krb5kdc[2155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164549, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
I20260808 04:49:09.081512 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.067234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:44423 (local address 127.4.217.62:37415)
0808 04:49:09.067389 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.067393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.068286 (+ 893us) server_negotiation.cc:407] Connection header received
0808 04:49:09.070172 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.070176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.070233 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.070316 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:09.071449 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.072282 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.074103 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.074528 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.074657 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.077681 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:09.077684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:09.077686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:09.077709 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:09.079230 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:09.079849 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:09.079851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:09.079852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:09.079974 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:09.080370 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:09.080372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:09.080374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:09.080545 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:09.080620 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.081281 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.081388 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:49:09.082439 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.067595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37415 (local address 127.4.217.1:44423)
0808 04:49:09.068133 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:09.068186 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:09.069916 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:09.070470 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:09.070476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:09.070830 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:09.071293 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.071305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.072449 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.072455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.073951 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.073983 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.074680 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.074683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.074816 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.075453 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:09.075470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:09.077502 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:09.079358 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:09.079367 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:09.079395 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:09.079742 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:09.080069 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:09.080074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:09.080076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:09.080274 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:09.080666 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:09.080671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:09.081044 (+ 373us) client_negotiation.cc:769] Sending connection context
0808 04:49:09.081206 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 04:49:09.081472 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:49:09.083772 2529 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37415
I20260808 04:49:09.084041 2529 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:09.084606 2529 heartbeater.cc:511] Master 127.4.217.62:37415 requested a full tablet report, sending...
I20260808 04:49:09.086195 2221 ts_manager.cc:194] Registered new tserver with Master: 88a28bc1176d4424a0fd3a278bc1a55e (127.4.217.1:46159)
I20260808 04:49:09.087327 2221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:44423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:49:09.099740 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37415
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:09.211380 2539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:09.211599 2539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:09.211656 2539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:09.215694 2539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:09.215745 2539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:09.215799 2539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:49:09.220731 2539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:09.221611 2539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:09.222510 2539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:09.231689 2550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:09.231709 2551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:09.231719 2555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:09.231954 2539 server_base.cc:1034] running on GCE node
I20260808 04:49:09.232344 2539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:09.233011 2539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:09.234205 2539 hybrid_clock.cc:648] HybridClock initialized: now 1786164549234190 us; error 24 us; skew 500 ppm
Aug 08 04:49:09 dist-test-slave-2kcd krb5kdc[2155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164549, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:09.237015 2539 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:49:09.237998 2539 webserver.cc:460] Webserver started at http://127.4.217.2:38261/ using document root <none> and password file <none>
I20260808 04:49:09.238556 2539 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:09.238648 2539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:09.238853 2539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:09.240871 2539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "e319feec41f2470aa92b2988be15364b"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "dc1db91d721d87413b5ba865412e12ef"
server_key_iv: "175a6ef231551c8e209d3767d6bdadb4"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.241631 2539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "e319feec41f2470aa92b2988be15364b"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "dc1db91d721d87413b5ba865412e12ef"
server_key_iv: "175a6ef231551c8e209d3767d6bdadb4"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.245517 2539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 04:49:09.247838 2565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.249090 2539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:49:09.249224 2539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "e319feec41f2470aa92b2988be15364b"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "dc1db91d721d87413b5ba865412e12ef"
server_key_iv: "175a6ef231551c8e209d3767d6bdadb4"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.249333 2539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:09.264993 2539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:09.267973 2539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:09.268177 2539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:09.268851 2539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:09.269718 2539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:09.269790 2539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.269855 2539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:09.269914 2539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.279170 2539 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:42583
I20260808 04:49:09.279192 2761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:42583 every 8 connection(s)
I20260808 04:49:09.280084 2539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:49:09.286113 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 2539
I20260808 04:49:09.286260 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:49:09.286583 4964 external_mini_cluster.cc:1428] Setting key f63793375837ad6b1171824f6b0438c5
Aug 08 04:49:09 dist-test-slave-2kcd krb5kdc[2155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164549, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:09.295090 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.282158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:42645 (local address 127.4.217.62:37415)
0808 04:49:09.282329 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.282334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.283248 (+ 914us) server_negotiation.cc:407] Connection header received
0808 04:49:09.284560 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.284562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.284612 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.284692 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:09.285946 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.286766 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.288095 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.288394 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.288508 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.291387 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:09.291391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:09.291395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:09.291427 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:09.292982 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:09.293514 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:09.293519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:09.293521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:09.293605 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:09.293954 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:09.293958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:09.293960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:09.294145 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:09.294258 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.294771 (+ 513us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.294915 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:49:09.295954 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.282607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37415 (local address 127.4.217.2:42645)
0808 04:49:09.283096 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:09.283139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:09.284346 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:09.284915 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:09.284922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:09.285288 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:09.285763 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.285777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.286900 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.286902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.287951 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.287963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.288556 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.288572 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.288722 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.289418 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:09.289435 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:09.291201 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:09.293117 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:09.293123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:09.293138 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:09.293402 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:09.293709 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:09.293712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:09.293714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:09.293857 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:09.294257 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:09.294266 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:09.294521 (+ 255us) client_negotiation.cc:769] Sending connection context
0808 04:49:09.294717 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 04:49:09.295049 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":157,"threads_started":1}
I20260808 04:49:09.297252 2762 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37415
I20260808 04:49:09.297531 2762 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
I20260808 04:49:09.298182 2762 heartbeater.cc:511] Master 127.4.217.62:37415 requested a full tablet report, sending...
I20260808 04:49:09.299317 2221 ts_manager.cc:194] Registered new tserver with Master: e319feec41f2470aa92b2988be15364b (127.4.217.2:42583)
I20260808 04:49:09.299996 2221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:42645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:49:09.318130 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37415
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:09.433293 2780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:09.433609 2780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:09.433691 2780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:09.438165 2780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:09.438267 2780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:09.438356 2780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:49:09.443769 2780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:09.444975 2780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:09.446080 2780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:09.456151 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:09.456151 2797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:09.456151 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:09.456382 2780 server_base.cc:1034] running on GCE node
I20260808 04:49:09.456943 2780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:09.457569 2780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:09.458761 2780 hybrid_clock.cc:648] HybridClock initialized: now 1786164549458747 us; error 25 us; skew 500 ppm
Aug 08 04:49:09 dist-test-slave-2kcd krb5kdc[2155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164549, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:09.461589 2780 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:49:09.462553 2780 webserver.cc:460] Webserver started at http://127.4.217.3:35563/ using document root <none> and password file <none>
I20260808 04:49:09.463125 2780 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:09.463186 2780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:09.463433 2780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:09.465577 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "149c6ad5009b4aedb25a3efdc07725e8"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "674f34eb80c62c24b25be5e2b53d4501"
server_key_iv: "9a8e83277b0be5c4694ecfa35ffadb60"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.466150 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "149c6ad5009b4aedb25a3efdc07725e8"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "674f34eb80c62c24b25be5e2b53d4501"
server_key_iv: "9a8e83277b0be5c4694ecfa35ffadb60"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.470052 2780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:49:09.472491 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.473675 2780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:09.473816 2780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "149c6ad5009b4aedb25a3efdc07725e8"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "674f34eb80c62c24b25be5e2b53d4501"
server_key_iv: "9a8e83277b0be5c4694ecfa35ffadb60"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.473928 2780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:09.490725 2780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:09.493846 2780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:09.494046 2780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:09.494617 2780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:09.495432 2780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:09.495503 2780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.495604 2780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:09.495656 2780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.504594 2780 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:37135
I20260808 04:49:09.504626 2986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:37135 every 8 connection(s)
I20260808 04:49:09.505529 2780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:49:09.515313 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 2780
I20260808 04:49:09.515463 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
Aug 08 04:49:09 dist-test-slave-2kcd krb5kdc[2155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164549, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:09.515846 4964 external_mini_cluster.cc:1428] Setting key 4d651ec1aaec060e9871cfc89f176f2b
I20260808 04:49:09.519654 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.507325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:53299 (local address 127.4.217.62:37415)
0808 04:49:09.507486 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.507491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.508555 (+ 1064us) server_negotiation.cc:407] Connection header received
0808 04:49:09.510527 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.510531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.510596 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.510696 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:09.511798 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.512344 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.513479 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.513698 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.513777 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.516352 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:09.516356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:09.516363 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:09.516392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:09.517935 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:09.518422 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:09.518425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:09.518426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:09.518495 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:09.518787 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:09.518789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:09.518790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:09.518947 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:09.519022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.519441 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.519540 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:49:09.520412 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.507836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:37415 (local address 127.4.217.3:53299)
0808 04:49:09.508406 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:09.508457 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:09.510222 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:09.510826 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:09.510833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:09.511192 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:09.511662 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.511674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.512454 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.512457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.513354 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.513363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.513826 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.513829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.513990 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.514481 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:09.514496 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:09.516189 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:09.518053 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:09.518058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:09.518073 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:09.518337 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:09.518582 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:09.518584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:09.518586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:09.518709 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:09.519036 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:09.519041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:09.519247 (+ 206us) client_negotiation.cc:769] Sending connection context
0808 04:49:09.519396 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:49:09.519636 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":177,"threads_started":1}
I20260808 04:49:09.521450 2987 heartbeater.cc:347] Connected to a master server at 127.4.217.62:37415
I20260808 04:49:09.521643 2987 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:09.522076 2987 heartbeater.cc:511] Master 127.4.217.62:37415 requested a full tablet report, sending...
I20260808 04:49:09.523000 2221 ts_manager.cc:194] Registered new tserver with Master: 149c6ad5009b4aedb25a3efdc07725e8 (127.4.217.3:37135)
I20260808 04:49:09.523521 2221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:53299
I20260808 04:49:09.529592 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:49:09.539237 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.531631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.4.217.62:37415)
0808 04:49:09.531781 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.531786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.531934 (+ 148us) server_negotiation.cc:407] Connection header received
0808 04:49:09.532100 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.532103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.532139 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.532221 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:09.532951 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.533411 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.534265 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.534477 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.534590 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.536033 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:09.536039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:09.536041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:09.536068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:09.537445 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:09.537975 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:09.537978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:09.537979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:09.538048 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:09.538386 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:09.538390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:09.538391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:09.538582 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:09.538736 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.538998 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.539085 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:49:09.542201 2221 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:49:09.545270 2221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:49:09.559717 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.553621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.4.217.1:46159)
0808 04:49:09.554717 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.554735 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.554757 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:49:09.554835 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.554841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.555038 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.555263 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:09.555703 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.556604 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.558291 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.559000 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.559126 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.559268 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.559457 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.559553 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":110,"threads_started":1}
I20260808 04:49:09.559724 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.553446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:46159 (local address 127.0.0.1:35116)
0808 04:49:09.554325 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:09.554341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:09.554432 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:09.555327 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:09.555333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:09.555370 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:09.555581 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.555589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.556728 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.556734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.558155 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.558167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.559130 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.559134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.559248 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.559287 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:49:09.559426 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 04:49:09.559545 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":50,"spinlock_wait_cycles":3584,"thread_start_us":108,"threads_started":1}
I20260808 04:49:09.559768 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.553436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:42583 (local address 127.0.0.1:60256)
0808 04:49:09.554119 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:09.554167 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:09.554292 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:09.555320 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:09.555327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:09.555362 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:09.555578 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.555587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.557056 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.557059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.558182 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.558189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.559320 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.559324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.559413 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.559437 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:49:09.559526 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 04:49:09.559600 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":24,"spinlock_wait_cycles":2432,"thread_start_us":95,"threads_started":1}
I20260808 04:49:09.560293 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.553718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:37135 (local address 127.0.0.1:43576)
0808 04:49:09.555447 (+ 1729us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:09.555463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:09.555572 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:09.556026 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:09.556029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:09.556048 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:09.556135 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.556141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.557438 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.557443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.558557 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.558568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.559766 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.559770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.559855 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.559875 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 04:49:09.559941 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 04:49:09.560001 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1671,"thread_start_us":107,"threads_started":1}
I20260808 04:49:09.560333 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.554280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.4.217.3:37135)
0808 04:49:09.554841 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.554852 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.555573 (+ 721us) server_negotiation.cc:407] Connection header received
0808 04:49:09.555682 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.555688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.555869 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.556022 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:09.556233 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.557258 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.558691 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.559484 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.559579 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.559706 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.560089 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.560184 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":99,"threads_started":1}
I20260808 04:49:09.560477 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.554238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.4.217.2:42583)
0808 04:49:09.554698 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.554712 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.554737 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:49:09.554814 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.554820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.555001 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.555178 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:09.555727 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.556919 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.558317 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.559014 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.559117 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.559253 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.560229 (+ 976us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.560313 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":165,"threads_started":1}
I20260808 04:49:09.562920 2426 tablet_service.cc:1467] Processing CreateTablet for tablet f14e096fd2b34ab0a61bfb94694b4d63 (DEFAULT_TABLE table=test-table [id=5db86d65980c46a79bf041482e6ea67d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:09.562917 2884 tablet_service.cc:1467] Processing CreateTablet for tablet f14e096fd2b34ab0a61bfb94694b4d63 (DEFAULT_TABLE table=test-table [id=5db86d65980c46a79bf041482e6ea67d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:09.564041 2426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14e096fd2b34ab0a61bfb94694b4d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:09.564044 2884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14e096fd2b34ab0a61bfb94694b4d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:09.566759 2648 tablet_service.cc:1467] Processing CreateTablet for tablet f14e096fd2b34ab0a61bfb94694b4d63 (DEFAULT_TABLE table=test-table [id=5db86d65980c46a79bf041482e6ea67d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:09.567688 2648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14e096fd2b34ab0a61bfb94694b4d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:09.571811 3015 tablet_bootstrap.cc:492] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e: Bootstrap starting.
I20260808 04:49:09.572923 3016 tablet_bootstrap.cc:492] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: Bootstrap starting.
I20260808 04:49:09.574146 3018 tablet_bootstrap.cc:492] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: Bootstrap starting.
I20260808 04:49:09.574429 3015 tablet_bootstrap.cc:654] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:09.574786 3016 tablet_bootstrap.cc:654] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:09.575814 3015 log.cc:824] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:09.576066 3016 log.cc:824] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:09.576721 3018 tablet_bootstrap.cc:654] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:09.577567 3018 log.cc:824] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:09.577948 3016 tablet_bootstrap.cc:492] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: No bootstrap required, opened a new log
I20260808 04:49:09.577996 3015 tablet_bootstrap.cc:492] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e: No bootstrap required, opened a new log
I20260808 04:49:09.578182 3016 ts_tablet_manager.cc:1397] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:49:09.578186 3015 ts_tablet_manager.cc:1397] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:49:09.579531 3018 tablet_bootstrap.cc:492] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: No bootstrap required, opened a new log
I20260808 04:49:09.579700 3018 ts_tablet_manager.cc:1397] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 04:49:09.581161 3016 raft_consensus.cc:348] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } }
I20260808 04:49:09.581396 3016 raft_consensus.cc:374] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:09.581451 3016 raft_consensus.cc:729] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149c6ad5009b4aedb25a3efdc07725e8, State: Initialized, Role: FOLLOWER
I20260808 04:49:09.581885 3016 consensus_queue.cc:260] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } }
I20260808 04:49:09.581938 3015 raft_consensus.cc:348] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } }
I20260808 04:49:09.582178 3015 raft_consensus.cc:374] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:09.582240 3015 raft_consensus.cc:729] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a28bc1176d4424a0fd3a278bc1a55e, State: Initialized, Role: FOLLOWER
I20260808 04:49:09.582640 2987 heartbeater.cc:503] Master 127.4.217.62:37415 was elected leader, sending a full tablet report...
I20260808 04:49:09.582684 3015 consensus_queue.cc:260] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } }
I20260808 04:49:09.583310 3018 raft_consensus.cc:348] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } }
I20260808 04:49:09.583540 3018 raft_consensus.cc:374] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:09.583647 3015 ts_tablet_manager.cc:1428] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260808 04:49:09.583678 3016 ts_tablet_manager.cc:1428] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:49:09.583976 2529 heartbeater.cc:503] Master 127.4.217.62:37415 was elected leader, sending a full tablet report...
I20260808 04:49:09.583598 3018 raft_consensus.cc:729] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e319feec41f2470aa92b2988be15364b, State: Initialized, Role: FOLLOWER
I20260808 04:49:09.585281 3018 consensus_queue.cc:260] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } }
I20260808 04:49:09.585878 2762 heartbeater.cc:503] Master 127.4.217.62:37415 was elected leader, sending a full tablet report...
I20260808 04:49:09.586222 3018 ts_tablet_manager.cc:1428] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260808 04:49:09.595082 3031 raft_consensus.cc:482] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:49:09.595252 3031 raft_consensus.cc:504] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } }
I20260808 04:49:09.596225 3031 leader_election.cc:288] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e319feec41f2470aa92b2988be15364b (127.4.217.2:42583), 88a28bc1176d4424a0fd3a278bc1a55e (127.4.217.1:46159)
I20260808 04:49:09.599910 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.596387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:42583 (local address 127.4.217.3:48627)
0808 04:49:09.596553 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:09.596570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:09.596655 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:09.597008 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:09.597012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:09.597035 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:09.597161 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.597170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.597942 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.597946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.598771 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.598780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.599470 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.599474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.599589 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.599635 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 04:49:09.599712 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:49:09.599769 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":3}
I20260808 04:49:09.599948 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.596451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:48627 (local address 127.4.217.2:42583)
0808 04:49:09.596589 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.596593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.596612 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:49:09.596811 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.596815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.596875 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.596955 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:09.597281 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.597835 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.598903 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.599352 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.599434 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.599483 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.599746 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.599823 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:49:09.600093 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.596451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:46159 (local address 127.4.217.3:57279)
0808 04:49:09.596831 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:09.596844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:09.596951 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:09.597197 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:09.597199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:09.597214 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:09.597354 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.597361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.598091 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.598093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.598921 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:09.598931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.599599 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.599602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:09.599775 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.599799 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:49:09.599875 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 04:49:09.599935 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":40,"thread_start_us":91,"threads_started":1}
I20260808 04:49:09.600143 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.596534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:57279 (local address 127.4.217.1:46159)
0808 04:49:09.596637 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.596641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.596989 (+ 348us) server_negotiation.cc:407] Connection header received
0808 04:49:09.597041 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.597044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.597086 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.597177 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:09.597468 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.597966 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.599045 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.599463 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.599568 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.599625 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.599927 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.600007 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 04:49:09.600469 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14e096fd2b34ab0a61bfb94694b4d63" candidate_uuid: "149c6ad5009b4aedb25a3efdc07725e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e319feec41f2470aa92b2988be15364b" is_pre_election: true
I20260808 04:49:09.600687 2454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14e096fd2b34ab0a61bfb94694b4d63" candidate_uuid: "149c6ad5009b4aedb25a3efdc07725e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" is_pre_election: true
I20260808 04:49:09.600724 2686 raft_consensus.cc:2454] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149c6ad5009b4aedb25a3efdc07725e8 in term 0.
I20260808 04:49:09.600965 2454 raft_consensus.cc:2454] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149c6ad5009b4aedb25a3efdc07725e8 in term 0.
I20260808 04:49:09.601245 2813 leader_election.cc:302] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149c6ad5009b4aedb25a3efdc07725e8, e319feec41f2470aa92b2988be15364b; no voters:
I20260808 04:49:09.601521 3031 raft_consensus.cc:2781] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:49:09.601671 3031 raft_consensus.cc:482] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:49:09.601764 3031 raft_consensus.cc:3037] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:09.602733 3031 raft_consensus.cc:504] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } }
I20260808 04:49:09.603093 3031 leader_election.cc:288] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [CANDIDATE]: Term 1 election: Requested vote from peers e319feec41f2470aa92b2988be15364b (127.4.217.2:42583), 88a28bc1176d4424a0fd3a278bc1a55e (127.4.217.1:46159)
I20260808 04:49:09.603418 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14e096fd2b34ab0a61bfb94694b4d63" candidate_uuid: "149c6ad5009b4aedb25a3efdc07725e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e319feec41f2470aa92b2988be15364b"
I20260808 04:49:09.603483 2454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14e096fd2b34ab0a61bfb94694b4d63" candidate_uuid: "149c6ad5009b4aedb25a3efdc07725e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a28bc1176d4424a0fd3a278bc1a55e"
I20260808 04:49:09.603551 2686 raft_consensus.cc:3037] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:09.603598 2454 raft_consensus.cc:3037] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:09.604519 2454 raft_consensus.cc:2454] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149c6ad5009b4aedb25a3efdc07725e8 in term 1.
I20260808 04:49:09.604523 2686 raft_consensus.cc:2454] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149c6ad5009b4aedb25a3efdc07725e8 in term 1.
I20260808 04:49:09.604836 2813 leader_election.cc:302] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149c6ad5009b4aedb25a3efdc07725e8, e319feec41f2470aa92b2988be15364b; no voters:
I20260808 04:49:09.605024 3031 raft_consensus.cc:2781] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:09.605232 3031 raft_consensus.cc:686] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 1 LEADER]: Becoming Leader. State: Replica: 149c6ad5009b4aedb25a3efdc07725e8, State: Running, Role: LEADER
I20260808 04:49:09.605564 3031 consensus_queue.cc:237] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } }
I20260808 04:49:09.608675 2221 catalog_manager.cc:5495] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149c6ad5009b4aedb25a3efdc07725e8 (127.4.217.3). New cstate: current_term: 1 leader_uuid: "149c6ad5009b4aedb25a3efdc07725e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149c6ad5009b4aedb25a3efdc07725e8" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37135 } health_report { overall_health: HEALTHY } } }
I20260808 04:49:09.622460 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.618727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.4.217.3:37135)
0808 04:49:09.618849 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.618854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.618965 (+ 111us) server_negotiation.cc:407] Connection header received
0808 04:49:09.619163 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.619166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.619216 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.619322 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:09.619571 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.620056 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.620976 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.621193 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.621303 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.621623 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:09.622004 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:09.622163 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.622265 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.622300 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:49:09.628378 2686 raft_consensus.cc:1261] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [term 1 FOLLOWER]: Refusing update from remote peer 149c6ad5009b4aedb25a3efdc07725e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:09.628399 2454 raft_consensus.cc:1261] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e [term 1 FOLLOWER]: Refusing update from remote peer 149c6ad5009b4aedb25a3efdc07725e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:09.629144 3031 consensus_queue.cc:1035] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a28bc1176d4424a0fd3a278bc1a55e" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:09.629360 3037 consensus_queue.cc:1035] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e319feec41f2470aa92b2988be15364b" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 42583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:09.637308 3045 mvcc.cc:205] Tried to move back new op lower bound from 7316129995270545408 to 7316129995187130368. Current Snapshot: MvccSnapshot[applied={T|T < 7316129995270545408}]
I20260808 04:49:09.637859 3050 mvcc.cc:205] Tried to move back new op lower bound from 7316129995270545408 to 7316129995187130368. Current Snapshot: MvccSnapshot[applied={T|T < 7316129995270545408}]
I20260808 04:49:09.639618 3046 mvcc.cc:205] Tried to move back new op lower bound from 7316129995270545408 to 7316129995187130368. Current Snapshot: MvccSnapshot[applied={T|T < 7316129995270545408}]
I20260808 04:49:09.647295 2220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:09.647455 2220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:09.650008 2220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3f83332ba3e44668cab0eaeb55a85c5: Sending DeleteTablet for 3 replicas of tablet f14e096fd2b34ab0a61bfb94694b4d63
I20260808 04:49:09.650828 2426 tablet_service.cc:1514] Processing DeleteTablet for tablet f14e096fd2b34ab0a61bfb94694b4d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:09 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:35116
I20260808 04:49:09.650938 2648 tablet_service.cc:1514] Processing DeleteTablet for tablet f14e096fd2b34ab0a61bfb94694b4d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:09 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:60256
I20260808 04:49:09.651062 2884 tablet_service.cc:1514] Processing DeleteTablet for tablet f14e096fd2b34ab0a61bfb94694b4d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:09 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:43576
I20260808 04:49:09.651563 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 2333
I20260808 04:49:09.651609 3059 tablet_replica.cc:330] T f14e096fd2b34ab0a61bfb94694b4d63 P 88a28bc1176d4424a0fd3a278bc1a55e: stopping tablet replica
I20260808 04:49:09.660300 3060 tablet_replica.cc:330] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: stopping tablet replica
I20260808 04:49:09.660681 3060 raft_consensus.cc:2229] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:09.661079 3060 raft_consensus.cc:2258] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:09.661482 3062 tablet_replica.cc:330] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: stopping tablet replica
I20260808 04:49:09.661866 3062 raft_consensus.cc:2229] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:49:09.662271 3062 raft_consensus.cc:2258] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:09.663527 3060 ts_tablet_manager.cc:1905] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:09.665926 3062 ts_tablet_manager.cc:1905] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:09.666481 3060 ts_tablet_manager.cc:1918] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:09.666584 3060 log.cc:1196] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/f14e096fd2b34ab0a61bfb94694b4d63
I20260808 04:49:09.666918 3060 ts_tablet_manager.cc:1939] T f14e096fd2b34ab0a61bfb94694b4d63 P e319feec41f2470aa92b2988be15364b: Deleting consensus metadata
I20260808 04:49:09.668210 2202 catalog_manager.cc:4841] TS e319feec41f2470aa92b2988be15364b (127.4.217.2:42583): tablet f14e096fd2b34ab0a61bfb94694b4d63 (table test-table [id=5db86d65980c46a79bf041482e6ea67d]) successfully deleted
W20260808 04:49:09.668720 2200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:46159 (error 108)
I20260808 04:49:09.668949 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 2539
I20260808 04:49:09.670006 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.669667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:46159 (local address 127.0.0.1:35128)
0808 04:49:09.669843 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:09.669905 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260808 04:49:09.669991 3062 ts_tablet_manager.cc:1918] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:09.670099 3062 log.cc:1196] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/f14e096fd2b34ab0a61bfb94694b4d63
W20260808 04:49:09.670280 2200 catalog_manager.cc:4584] TS 88a28bc1176d4424a0fd3a278bc1a55e (127.4.217.1:46159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f14e096fd2b34ab0a61bfb94694b4d63: Network error: Client connection negotiation failed: client connection to 127.4.217.1:46159: connect: Connection refused (error 111)
I20260808 04:49:09.670491 3062 ts_tablet_manager.cc:1939] T f14e096fd2b34ab0a61bfb94694b4d63 P 149c6ad5009b4aedb25a3efdc07725e8: Deleting consensus metadata
I20260808 04:49:09.676505 2199 catalog_manager.cc:4841] TS 149c6ad5009b4aedb25a3efdc07725e8 (127.4.217.3:37135): tablet f14e096fd2b34ab0a61bfb94694b4d63 (table test-table [id=5db86d65980c46a79bf041482e6ea67d]) successfully deleted
I20260808 04:49:09.683569 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 2780
I20260808 04:49:09.697808 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 2174
2026-08-08T04:49:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2489 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T04:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:09Z Disabled control of system clock
I20260808 04:49:09.736965 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:38227
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:49:09.851619 3069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:09.851871 3069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:09.851904 3069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:09.855875 3069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:09.855933 3069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:09.855949 3069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:09.855966 3069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:09.855981 3069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:09.860536 3069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:38227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:09.861570 3069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:09.862969 3069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:09.871451 3080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:09.871428 3079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:09.871569 3069 server_base.cc:1034] running on GCE node
W20260808 04:49:09.871436 3082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:09.872316 3069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:09.873442 3069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:09.874630 3069 hybrid_clock.cc:648] HybridClock initialized: now 1786164549874600 us; error 41 us; skew 500 ppm
I20260808 04:49:09.876601 3069 webserver.cc:460] Webserver started at http://127.4.217.62:40151/ using document root <none> and password file <none>
I20260808 04:49:09.877321 3069 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:09.877388 3069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:09.877552 3069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:09.879478 3069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "391e90d10508424f9879616d82279233"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "be958123fd614344f5ccaccdd609cd27"
server_key_iv: "7cdde2985aec6fb1929251f9ac6b0f32"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.879971 3069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "391e90d10508424f9879616d82279233"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "be958123fd614344f5ccaccdd609cd27"
server_key_iv: "7cdde2985aec6fb1929251f9ac6b0f32"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.883502 3069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:49:09.885747 3089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:09.886860 3069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:09.886981 3069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "391e90d10508424f9879616d82279233"
format_stamp: "Formatted at 2026-08-08 04:49:09 on dist-test-slave-2kcd"
server_key: "be958123fd614344f5ccaccdd609cd27"
server_key_iv: "7cdde2985aec6fb1929251f9ac6b0f32"
server_key_version: "encryptionkey@0"
I20260808 04:49:09.887059 3069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:09.903301 3069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:09.903841 3069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:09.903990 3069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:09.910984 3069 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:38227
I20260808 04:49:09.911012 3187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:38227 every 8 connection(s)
I20260808 04:49:09.911953 3069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:09.913125 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 3069
I20260808 04:49:09.913277 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:09.913565 4964 external_mini_cluster.cc:1428] Setting key 94bfab09d74b696edfe686e7fc23e70d
I20260808 04:49:09.915457 3188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:09.920663 3188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233: Bootstrap starting.
I20260808 04:49:09.922430 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:09.914882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.4.217.62:38227)
0808 04:49:09.915515 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:09.915530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:09.915592 (+ 62us) server_negotiation.cc:407] Connection header received
0808 04:49:09.916235 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:09.916251 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:09.916616 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:09.916993 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:09.917396 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.918334 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.919246 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:09.919698 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:09.919874 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:09.920763 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:09.920788 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:09.920823 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:09.921200 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:09.921252 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:09.921272 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:09.921386 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:09.921502 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 04:49:09.921772 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":157,"threads_started":1}
I20260808 04:49:09.922814 3188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:09.923530 3188 log.cc:824] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:09.925282 3188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233: No bootstrap required, opened a new log
I20260808 04:49:09.928503 3188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391e90d10508424f9879616d82279233" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38227 } }
I20260808 04:49:09.928750 3188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:09.928879 3188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391e90d10508424f9879616d82279233, State: Initialized, Role: FOLLOWER
I20260808 04:49:09.929286 3188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391e90d10508424f9879616d82279233" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38227 } }
I20260808 04:49:09.929426 3188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:09.929508 3188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:09.929630 3188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:09.930433 3188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391e90d10508424f9879616d82279233" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38227 } }
I20260808 04:49:09.930764 3188 leader_election.cc:302] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391e90d10508424f9879616d82279233; no voters:
I20260808 04:49:09.931049 3188 leader_election.cc:288] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:09.931161 3197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:09.931423 3197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [term 1 LEADER]: Becoming Leader. State: Replica: 391e90d10508424f9879616d82279233, State: Running, Role: LEADER
I20260808 04:49:09.931692 3197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391e90d10508424f9879616d82279233" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38227 } }
I20260808 04:49:09.932212 3188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:09.933235 3199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "391e90d10508424f9879616d82279233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391e90d10508424f9879616d82279233" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38227 } } }
I20260808 04:49:09.933374 3199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:09.933760 3209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:09.933728 3200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391e90d10508424f9879616d82279233. Latest consensus state: current_term: 1 leader_uuid: "391e90d10508424f9879616d82279233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391e90d10508424f9879616d82279233" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38227 } } }
I20260808 04:49:09.934042 3200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:09.936930 3209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:09.942479 3209 catalog_manager.cc:1347] Generated new cluster ID: eca44c6746a946f7aa4c7defa75d6220
I20260808 04:49:09.942541 3209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:09.949529 3209 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:09.950363 3209 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:09.956801 3209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233: Generated new TSK 0
I20260808 04:49:09.957443 3209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:09.965865 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38227
--builtin_ntp_servers=127.4.217.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:49:10.079825 3231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:10.080055 3231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:10.080116 3231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:10.084134 3231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:10.084185 3231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:10.084242 3231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:49:10.088967 3231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:10.090006 3231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:10.091442 3231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:10.100920 3240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:10.101020 3237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:10.101027 3243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:10.101217 3231 server_base.cc:1034] running on GCE node
I20260808 04:49:10.101647 3231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:10.102272 3231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:10.103458 3231 hybrid_clock.cc:648] HybridClock initialized: now 1786164550103430 us; error 39 us; skew 500 ppm
I20260808 04:49:10.105479 3231 webserver.cc:460] Webserver started at http://127.4.217.1:46799/ using document root <none> and password file <none>
I20260808 04:49:10.106068 3231 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:10.106165 3231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:10.106405 3231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:10.108404 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "0ea7f0f44f874b1c80c7cac11debfccb"
format_stamp: "Formatted at 2026-08-08 04:49:10 on dist-test-slave-2kcd"
server_key: "1a3d6db795abc1cb727ee080e9c7238e"
server_key_iv: "dbac267471936ef09edf587e6c6ec706"
server_key_version: "encryptionkey@0"
I20260808 04:49:10.109038 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "0ea7f0f44f874b1c80c7cac11debfccb"
format_stamp: "Formatted at 2026-08-08 04:49:10 on dist-test-slave-2kcd"
server_key: "1a3d6db795abc1cb727ee080e9c7238e"
server_key_iv: "dbac267471936ef09edf587e6c6ec706"
server_key_version: "encryptionkey@0"
I20260808 04:49:10.112490 3231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:49:10.114897 3250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:10.116029 3231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:10.116166 3231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "0ea7f0f44f874b1c80c7cac11debfccb"
format_stamp: "Formatted at 2026-08-08 04:49:10 on dist-test-slave-2kcd"
server_key: "1a3d6db795abc1cb727ee080e9c7238e"
server_key_iv: "dbac267471936ef09edf587e6c6ec706"
server_key_version: "encryptionkey@0"
I20260808 04:49:10.116281 3231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:10.125584 3231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:10.126183 3231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:10.126379 3231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:10.126945 3231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:10.127820 3231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:10.127893 3231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:10.127974 3231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:10.128010 3231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:10.137146 3231 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:42177
I20260808 04:49:10.137177 3439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:42177 every 8 connection(s)
I20260808 04:49:10.138012 3231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:49:10.143242 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 3231
I20260808 04:49:10.143389 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:49:10.143682 4964 external_mini_cluster.cc:1428] Setting key 3017479dbf81ebe15854caaac3ed09a4
I20260808 04:49:10.145617 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38227
--builtin_ntp_servers=127.4.217.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:49:10.147837 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.139756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:53941 (local address 127.4.217.62:38227)
0808 04:49:10.139897 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.139909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.140967 (+ 1058us) server_negotiation.cc:407] Connection header received
0808 04:49:10.142458 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.142462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.142529 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.142636 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:10.143566 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.144258 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.145605 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.145942 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.146060 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.147172 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:10.147176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:10.147178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:10.147196 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:10.147223 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:10.147237 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:10.147318 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.147549 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.147641 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:49:10.148602 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.140129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38227 (local address 127.4.217.1:53941)
0808 04:49:10.140749 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:10.140836 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:10.142206 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:10.142810 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:10.142817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:10.142897 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:10.143394 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.143414 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.144397 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.144402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.145478 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.145488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.146099 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.146103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.146270 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.146907 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:49:10.146909 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:10.146949 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:49:10.147015 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:10.147362 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:10.147376 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:10.147385 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:49:10.147524 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 04:49:10.147756 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260808 04:49:10.149950 3440 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38227
I20260808 04:49:10.150200 3440 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:10.150839 3440 heartbeater.cc:511] Master 127.4.217.62:38227 requested a full tablet report, sending...
I20260808 04:49:10.152422 3115 ts_manager.cc:194] Registered new tserver with Master: 0ea7f0f44f874b1c80c7cac11debfccb (127.4.217.1:42177)
I20260808 04:49:10.153661 3115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.1:53941
W20260808 04:49:10.262403 3444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:10.262651 3444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:10.262712 3444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:10.266659 3444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:10.266705 3444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:10.266788 3444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:49:10.271633 3444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:10.272653 3444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:10.273985 3444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:10.283303 3451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:10.283388 3452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:10.283459 3444 server_base.cc:1034] running on GCE node
W20260808 04:49:10.283361 3454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:10.283901 3444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:10.284530 3444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:10.285710 3444 hybrid_clock.cc:648] HybridClock initialized: now 1786164550285679 us; error 45 us; skew 500 ppm
I20260808 04:49:10.287420 3444 webserver.cc:460] Webserver started at http://127.4.217.2:33133/ using document root <none> and password file <none>
I20260808 04:49:10.287964 3444 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:10.288064 3444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:10.288291 3444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:10.290396 3444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "553d5cb11c48493c87665011f6cebdee"
format_stamp: "Formatted at 2026-08-08 04:49:10 on dist-test-slave-2kcd"
server_key: "2b184d98b15f192f8dfa21f86aaab14b"
server_key_iv: "d46f4bbf22dae3750c44979b8e927d10"
server_key_version: "encryptionkey@0"
I20260808 04:49:10.290948 3444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "553d5cb11c48493c87665011f6cebdee"
format_stamp: "Formatted at 2026-08-08 04:49:10 on dist-test-slave-2kcd"
server_key: "2b184d98b15f192f8dfa21f86aaab14b"
server_key_iv: "d46f4bbf22dae3750c44979b8e927d10"
server_key_version: "encryptionkey@0"
I20260808 04:49:10.294492 3444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:49:10.296733 3463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:10.298005 3444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:10.298149 3444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "553d5cb11c48493c87665011f6cebdee"
format_stamp: "Formatted at 2026-08-08 04:49:10 on dist-test-slave-2kcd"
server_key: "2b184d98b15f192f8dfa21f86aaab14b"
server_key_iv: "d46f4bbf22dae3750c44979b8e927d10"
server_key_version: "encryptionkey@0"
I20260808 04:49:10.298255 3444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:10.310031 3444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:10.310623 3444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:10.310807 3444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:10.311353 3444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:10.312249 3444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:10.312320 3444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:10.312389 3444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:10.312443 3444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:10.320994 3444 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:41153
I20260808 04:49:10.321076 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:41153 every 8 connection(s)
I20260808 04:49:10.321868 3444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:49:10.323388 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 3444
I20260808 04:49:10.323553 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:49:10.323941 4964 external_mini_cluster.cc:1428] Setting key 013267b29b753305a7d00bd240809b61
I20260808 04:49:10.326120 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38227
--builtin_ntp_servers=127.4.217.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:49:10.332008 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.323757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:60891 (local address 127.4.217.62:38227)
0808 04:49:10.323916 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.323921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.325010 (+ 1089us) server_negotiation.cc:407] Connection header received
0808 04:49:10.326749 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.326752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.326805 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.326893 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:10.327755 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.328605 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.329797 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.330195 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.330323 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.331416 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:10.331422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:10.331426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:10.331447 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:10.331472 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:10.331486 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:10.331575 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.331723 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.331799 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:49:10.332695 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.324142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38227 (local address 127.4.217.2:60891)
0808 04:49:10.324850 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:10.324911 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:10.326513 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:10.327064 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:10.327071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:10.327151 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:10.327611 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.327623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.328744 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.328747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.329666 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.329676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.330345 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.330347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.330477 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.331143 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:49:10.331145 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:10.331183 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:49:10.331243 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:10.331579 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:10.331585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:10.331589 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:49:10.331682 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:49:10.331883 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":115,"threads_started":1}
I20260808 04:49:10.334151 3643 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38227
I20260808 04:49:10.334457 3643 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:10.334971 3643 heartbeater.cc:511] Master 127.4.217.62:38227 requested a full tablet report, sending...
I20260808 04:49:10.336110 3115 ts_manager.cc:194] Registered new tserver with Master: 553d5cb11c48493c87665011f6cebdee (127.4.217.2:41153)
I20260808 04:49:10.337019 3115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.2:60891
W20260808 04:49:10.440217 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:10.440474 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:10.440513 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:10.444419 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:10.444476 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:10.444530 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:49:10.449297 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:10.450317 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:10.451663 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:10.461467 3653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:10.461475 3656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:10.461690 3646 server_base.cc:1034] running on GCE node
W20260808 04:49:10.461490 3652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:10.462159 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:10.462881 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:10.464066 3646 hybrid_clock.cc:648] HybridClock initialized: now 1786164550464036 us; error 42 us; skew 500 ppm
I20260808 04:49:10.465991 3646 webserver.cc:460] Webserver started at http://127.4.217.3:36769/ using document root <none> and password file <none>
I20260808 04:49:10.466552 3646 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:10.466610 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:10.466777 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:10.469061 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "3725c1f6699a46eeb21d15fda5d75777"
format_stamp: "Formatted at 2026-08-08 04:49:10 on dist-test-slave-2kcd"
server_key: "bd9b8b7555c59c7b59833e142685cfc7"
server_key_iv: "63f7adf2e6de8940a43cd0b5f06c1005"
server_key_version: "encryptionkey@0"
I20260808 04:49:10.469632 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "3725c1f6699a46eeb21d15fda5d75777"
format_stamp: "Formatted at 2026-08-08 04:49:10 on dist-test-slave-2kcd"
server_key: "bd9b8b7555c59c7b59833e142685cfc7"
server_key_iv: "63f7adf2e6de8940a43cd0b5f06c1005"
server_key_version: "encryptionkey@0"
I20260808 04:49:10.473254 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:49:10.475646 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:10.476914 3646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:10.477012 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "3725c1f6699a46eeb21d15fda5d75777"
format_stamp: "Formatted at 2026-08-08 04:49:10 on dist-test-slave-2kcd"
server_key: "bd9b8b7555c59c7b59833e142685cfc7"
server_key_iv: "63f7adf2e6de8940a43cd0b5f06c1005"
server_key_version: "encryptionkey@0"
I20260808 04:49:10.477095 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:10.490592 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:10.491282 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:10.491441 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:10.491966 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:10.492936 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:10.492987 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:10.493028 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:10.493044 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:10.502710 3646 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:37793
I20260808 04:49:10.502743 3846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:37793 every 8 connection(s)
I20260808 04:49:10.503679 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:49:10.513746 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 3646
I20260808 04:49:10.513863 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:49:10.513994 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.505584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:34707 (local address 127.4.217.62:38227)
0808 04:49:10.505726 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.505730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.506984 (+ 1254us) server_negotiation.cc:407] Connection header received
0808 04:49:10.508347 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.508350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.508401 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.508484 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:10.509460 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.509985 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.511209 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.511583 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.511701 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.513193 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:10.513198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:10.513200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:10.513217 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:10.513240 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:10.513250 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:10.513297 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.513547 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.513643 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:49:10.514271 4964 external_mini_cluster.cc:1428] Setting key 97b1a15f7fefb65173a9143e0cafe5ed
I20260808 04:49:10.514801 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.506147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38227 (local address 127.4.217.3:34707)
0808 04:49:10.506816 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:10.506855 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:10.508137 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:10.508661 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:10.508668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:10.508781 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:10.509323 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.509336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.510108 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.510110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.510989 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.510999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.511756 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.511763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.512008 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.512812 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:49:10.512816 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:10.512870 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:49:10.512975 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:10.513371 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:10.513380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:10.513386 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:49:10.513540 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 04:49:10.513822 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
I20260808 04:49:10.515933 3847 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38227
I20260808 04:49:10.516182 3847 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:10.516837 3847 heartbeater.cc:511] Master 127.4.217.62:38227 requested a full tablet report, sending...
I20260808 04:49:10.517822 3115 ts_manager.cc:194] Registered new tserver with Master: 3725c1f6699a46eeb21d15fda5d75777 (127.4.217.3:37793)
I20260808 04:49:10.518391 3115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.3:34707
I20260808 04:49:10.528477 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:49:10.534641 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.530511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.4.217.62:38227)
0808 04:49:10.530657 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.530662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.530839 (+ 177us) server_negotiation.cc:407] Connection header received
0808 04:49:10.530995 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.530998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.531058 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.531140 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:10.531412 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.531918 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.532916 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.533174 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.533282 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.534129 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:10.534132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:10.534135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:10.534147 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:10.534172 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:10.534180 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:10.534336 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.534463 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.534497 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:49:10.537243 3115 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:49:10.540226 3115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:49:10.555446 3879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.549971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.4.217.1:42177)
0808 04:49:10.550477 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.550488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.550853 (+ 365us) server_negotiation.cc:407] Connection header received
0808 04:49:10.550937 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.550942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.551126 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.551290 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:10.551470 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.552196 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.554139 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.554727 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.554856 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.555062 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.555167 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.555267 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":113,"threads_started":1}
I20260808 04:49:10.555440 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.549179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:41153 (local address 127.0.0.1:58400)
0808 04:49:10.549949 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:10.549962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:10.550053 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:10.550727 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:10.550736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:10.550768 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:10.550986 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.550996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.552516 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.552519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.553713 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.553724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.554878 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.554883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.555061 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.555100 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:49:10.555165 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 04:49:10.555280 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":128,"thread_start_us":90,"threads_started":1}
I20260808 04:49:10.555816 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.548982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:37793 (local address 127.0.0.1:42372)
0808 04:49:10.549502 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:10.549550 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:10.549683 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:10.550810 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:10.550813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:10.550826 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:10.551025 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.551035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.552457 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.552460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.553846 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.553856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.554879 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.554883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.555058 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.555218 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 04:49:10.555267 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 04:49:10.555333 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":254,"thread_start_us":74,"threads_started":1}
I20260808 04:49:10.555822 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.549109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.4.217.3:37793)
0808 04:49:10.550333 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.550347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.550372 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:49:10.550450 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.550454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.550656 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.550965 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:10.551128 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.552345 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.553973 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.554734 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.555130 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.555339 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.555390 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.555484 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1108,"thread_start_us":144,"threads_started":1}
I20260808 04:49:10.556224 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.549792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:42177 (local address 127.0.0.1:50078)
0808 04:49:10.550660 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:10.550673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:10.550756 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:10.551270 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:10.551273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:10.551292 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:10.551374 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.551379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.552318 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.552321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.554005 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.554012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.554858 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.554862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.554966 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.555003 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:49:10.555138 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 04:49:10.556104 (+ 966us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":271,"thread_start_us":101,"threads_started":1}
I20260808 04:49:10.556330 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.549680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.4.217.2:41153)
0808 04:49:10.550183 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.550195 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.550222 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:49:10.550307 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.550316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.550539 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.550711 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:10.551128 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.552345 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.553843 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.554727 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.555479 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.555682 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.555737 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.555824 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":146,"threads_started":1}
I20260808 04:49:10.559113 3741 tablet_service.cc:1467] Processing CreateTablet for tablet 435d025c029a487d8927656ddd6ba12e (DEFAULT_TABLE table=test-table [id=f5e301dd25174f20bef89920222fb9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:10.559214 3327 tablet_service.cc:1467] Processing CreateTablet for tablet 435d025c029a487d8927656ddd6ba12e (DEFAULT_TABLE table=test-table [id=f5e301dd25174f20bef89920222fb9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:10.559213 3535 tablet_service.cc:1467] Processing CreateTablet for tablet 435d025c029a487d8927656ddd6ba12e (DEFAULT_TABLE table=test-table [id=f5e301dd25174f20bef89920222fb9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:10.560222 3741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435d025c029a487d8927656ddd6ba12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:10.560235 3535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435d025c029a487d8927656ddd6ba12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:10.560235 3327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435d025c029a487d8927656ddd6ba12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:10.566745 3880 tablet_bootstrap.cc:492] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: Bootstrap starting.
I20260808 04:49:10.568130 3881 tablet_bootstrap.cc:492] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: Bootstrap starting.
I20260808 04:49:10.569159 3882 tablet_bootstrap.cc:492] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb: Bootstrap starting.
I20260808 04:49:10.569533 3880 tablet_bootstrap.cc:654] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:10.570268 3880 log.cc:824] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:10.570361 3881 tablet_bootstrap.cc:654] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:10.571184 3882 tablet_bootstrap.cc:654] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:10.571184 3881 log.cc:824] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:10.571980 3882 log.cc:824] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:10.572748 3880 tablet_bootstrap.cc:492] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: No bootstrap required, opened a new log
I20260808 04:49:10.572976 3880 ts_tablet_manager.cc:1397] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260808 04:49:10.573144 3881 tablet_bootstrap.cc:492] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: No bootstrap required, opened a new log
I20260808 04:49:10.573325 3881 ts_tablet_manager.cc:1397] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:49:10.574020 3882 tablet_bootstrap.cc:492] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb: No bootstrap required, opened a new log
I20260808 04:49:10.574198 3882 ts_tablet_manager.cc:1397] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:49:10.576371 3880 raft_consensus.cc:348] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } }
I20260808 04:49:10.576620 3880 raft_consensus.cc:374] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:10.576674 3880 raft_consensus.cc:729] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3725c1f6699a46eeb21d15fda5d75777, State: Initialized, Role: FOLLOWER
I20260808 04:49:10.577158 3880 consensus_queue.cc:260] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } }
I20260808 04:49:10.577600 3881 raft_consensus.cc:348] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } }
I20260808 04:49:10.577848 3881 raft_consensus.cc:374] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:10.577895 3881 raft_consensus.cc:729] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553d5cb11c48493c87665011f6cebdee, State: Initialized, Role: FOLLOWER
I20260808 04:49:10.578367 3847 heartbeater.cc:503] Master 127.4.217.62:38227 was elected leader, sending a full tablet report...
I20260808 04:49:10.578351 3881 consensus_queue.cc:260] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } }
I20260808 04:49:10.578395 3882 raft_consensus.cc:348] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } }
I20260808 04:49:10.578632 3882 raft_consensus.cc:374] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:10.578666 3880 ts_tablet_manager.cc:1428] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260808 04:49:10.578683 3882 raft_consensus.cc:729] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea7f0f44f874b1c80c7cac11debfccb, State: Initialized, Role: FOLLOWER
I20260808 04:49:10.579089 3882 consensus_queue.cc:260] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } }
I20260808 04:49:10.579288 3881 ts_tablet_manager.cc:1428] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:49:10.579433 3643 heartbeater.cc:503] Master 127.4.217.62:38227 was elected leader, sending a full tablet report...
I20260808 04:49:10.579991 3882 ts_tablet_manager.cc:1428] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:49:10.580281 3440 heartbeater.cc:503] Master 127.4.217.62:38227 was elected leader, sending a full tablet report...
W20260808 04:49:10.639199 3441 tablet.cc:2369] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:49:10.690685 3893 raft_consensus.cc:482] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:49:10.690965 3893 raft_consensus.cc:504] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } }
I20260808 04:49:10.692268 3893 leader_election.cc:288] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3725c1f6699a46eeb21d15fda5d75777 (127.4.217.3:37793), 553d5cb11c48493c87665011f6cebdee (127.4.217.2:41153)
I20260808 04:49:10.697980 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.692561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:41153 (local address 127.4.217.1:34743)
0808 04:49:10.693000 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:10.693011 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:10.693093 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:10.693354 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:10.693357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:10.693398 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:10.693708 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.693716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.694750 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.694753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.696485 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.696496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.697363 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.697367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.697616 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.697660 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 04:49:10.697737 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:49:10.697817 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":408,"spinlock_wait_cycles":31360,"thread_start_us":81,"threads_started":1}
I20260808 04:49:10.697985 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.692674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:42737 (local address 127.4.217.3:37793)
0808 04:49:10.692964 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.692969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.692989 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:49:10.693184 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.693187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.693241 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.693330 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:10.693831 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.694640 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.696614 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.697228 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.697331 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.697394 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.697789 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.697880 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260808 04:49:10.698010 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.692798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:34743 (local address 127.4.217.2:41153)
0808 04:49:10.692962 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.692967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.693086 (+ 119us) server_negotiation.cc:407] Connection header received
0808 04:49:10.693181 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.693184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.693241 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.693330 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:10.693837 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.694628 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.696616 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.697229 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.697331 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.697394 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.697779 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.697862 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:49:10.698560 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.692561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:37793 (local address 127.4.217.1:42737)
0808 04:49:10.692908 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:10.692925 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:10.693058 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:10.693354 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:10.693357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:10.693400 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:10.693711 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.693716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.694750 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.694753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.696485 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:10.696496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.697363 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.697367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:10.697613 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.697659 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 04:49:10.698316 (+ 657us) client_negotiation.cc:240] Negotiation successful
0808 04:49:10.698394 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":389,"spinlock_wait_cycles":7040,"thread_start_us":117,"threads_started":1}
I20260808 04:49:10.699163 3564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435d025c029a487d8927656ddd6ba12e" candidate_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553d5cb11c48493c87665011f6cebdee" is_pre_election: true
I20260808 04:49:10.699180 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435d025c029a487d8927656ddd6ba12e" candidate_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725c1f6699a46eeb21d15fda5d75777" is_pre_election: true
I20260808 04:49:10.699430 3564 raft_consensus.cc:2454] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea7f0f44f874b1c80c7cac11debfccb in term 0.
I20260808 04:49:10.699474 3772 raft_consensus.cc:2454] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea7f0f44f874b1c80c7cac11debfccb in term 0.
I20260808 04:49:10.699891 3255 leader_election.cc:302] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea7f0f44f874b1c80c7cac11debfccb, 553d5cb11c48493c87665011f6cebdee; no voters:
I20260808 04:49:10.700213 3893 raft_consensus.cc:2781] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:49:10.700302 3893 raft_consensus.cc:482] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:49:10.700348 3893 raft_consensus.cc:3037] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:10.701526 3893 raft_consensus.cc:504] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } }
I20260808 04:49:10.701840 3893 leader_election.cc:288] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [CANDIDATE]: Term 1 election: Requested vote from peers 3725c1f6699a46eeb21d15fda5d75777 (127.4.217.3:37793), 553d5cb11c48493c87665011f6cebdee (127.4.217.2:41153)
I20260808 04:49:10.702114 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435d025c029a487d8927656ddd6ba12e" candidate_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725c1f6699a46eeb21d15fda5d75777"
I20260808 04:49:10.702241 3772 raft_consensus.cc:3037] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:10.702381 3564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435d025c029a487d8927656ddd6ba12e" candidate_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553d5cb11c48493c87665011f6cebdee"
I20260808 04:49:10.702503 3564 raft_consensus.cc:3037] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:10.703537 3772 raft_consensus.cc:2454] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea7f0f44f874b1c80c7cac11debfccb in term 1.
I20260808 04:49:10.703838 3564 raft_consensus.cc:2454] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea7f0f44f874b1c80c7cac11debfccb in term 1.
I20260808 04:49:10.703859 3258 leader_election.cc:302] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea7f0f44f874b1c80c7cac11debfccb, 3725c1f6699a46eeb21d15fda5d75777; no voters:
I20260808 04:49:10.704099 3893 raft_consensus.cc:2781] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:10.704315 3893 raft_consensus.cc:686] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [term 1 LEADER]: Becoming Leader. State: Replica: 0ea7f0f44f874b1c80c7cac11debfccb, State: Running, Role: LEADER
I20260808 04:49:10.704668 3893 consensus_queue.cc:237] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } }
I20260808 04:49:10.707773 3113 catalog_manager.cc:5495] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ea7f0f44f874b1c80c7cac11debfccb (127.4.217.1). New cstate: current_term: 1 leader_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea7f0f44f874b1c80c7cac11debfccb" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42177 } health_report { overall_health: HEALTHY } } }
I20260808 04:49:10.744242 3879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.740883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.4.217.1:42177)
0808 04:49:10.741026 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:10.741029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:10.741043 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:49:10.741116 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:10.741118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:10.741174 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:10.741277 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:10.741526 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.742053 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.742923 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:10.743123 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:10.743197 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:10.743449 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:10.743864 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:10.743982 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:10.744098 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 04:49:10.744133 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:49:10.750373 3772 raft_consensus.cc:1261] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [term 1 FOLLOWER]: Refusing update from remote peer 0ea7f0f44f874b1c80c7cac11debfccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:10.750360 3564 raft_consensus.cc:1261] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [term 1 FOLLOWER]: Refusing update from remote peer 0ea7f0f44f874b1c80c7cac11debfccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:10.751130 3893 consensus_queue.cc:1035] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725c1f6699a46eeb21d15fda5d75777" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:10.751366 3899 consensus_queue.cc:1035] T 435d025c029a487d8927656ddd6ba12e P 0ea7f0f44f874b1c80c7cac11debfccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "553d5cb11c48493c87665011f6cebdee" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 41153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:49:10.754568 3849 tablet.cc:2369] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:49:10.758657 3903 mvcc.cc:205] Tried to move back new op lower bound from 7316129999865577472 to 7316129999689166848. Current Snapshot: MvccSnapshot[applied={T|T < 7316129999865577472}]
I20260808 04:49:10.759554 3910 mvcc.cc:205] Tried to move back new op lower bound from 7316129999865577472 to 7316129999689166848. Current Snapshot: MvccSnapshot[applied={T|T < 7316129999865577472}]
I20260808 04:49:10.760746 3905 mvcc.cc:205] Tried to move back new op lower bound from 7316129999865577472 to 7316129999689166848. Current Snapshot: MvccSnapshot[applied={T|T < 7316129999865577472}]
I20260808 04:49:10.768615 3112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:10.768759 3112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:10.771131 3112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 391e90d10508424f9879616d82279233: Sending DeleteTablet for 3 replicas of tablet 435d025c029a487d8927656ddd6ba12e
I20260808 04:49:10.771966 3327 tablet_service.cc:1514] Processing DeleteTablet for tablet 435d025c029a487d8927656ddd6ba12e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:10 UTC) from {username='slave'} at 127.0.0.1:50078
I20260808 04:49:10.771997 3741 tablet_service.cc:1514] Processing DeleteTablet for tablet 435d025c029a487d8927656ddd6ba12e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:10 UTC) from {username='slave'} at 127.0.0.1:42372
I20260808 04:49:10.772229 3535 tablet_service.cc:1514] Processing DeleteTablet for tablet 435d025c029a487d8927656ddd6ba12e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:10 UTC) from {username='slave'} at 127.0.0.1:58400
I20260808 04:49:10.772554 3925 tablet_replica.cc:330] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: stopping tablet replica
I20260808 04:49:10.772686 3926 tablet_replica.cc:330] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: stopping tablet replica
I20260808 04:49:10.773010 3926 raft_consensus.cc:2229] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:10.773005 3925 raft_consensus.cc:2229] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:10.773370 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 3231
I20260808 04:49:10.776119 3926 raft_consensus.cc:2258] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:10.773538 3925 raft_consensus.cc:2258] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:10.782603 3925 ts_tablet_manager.cc:1905] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:10.785084 3925 ts_tablet_manager.cc:1918] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:10.785197 3925 log.cc:1196] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/435d025c029a487d8927656ddd6ba12e
I20260808 04:49:10.785509 3925 ts_tablet_manager.cc:1939] T 435d025c029a487d8927656ddd6ba12e P 3725c1f6699a46eeb21d15fda5d75777: Deleting consensus metadata
I20260808 04:49:10.785573 3926 ts_tablet_manager.cc:1905] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:10.786778 3097 catalog_manager.cc:4841] TS 3725c1f6699a46eeb21d15fda5d75777 (127.4.217.3:37793): tablet 435d025c029a487d8927656ddd6ba12e (table test-table [id=f5e301dd25174f20bef89920222fb9cf]) successfully deleted
I20260808 04:49:10.789469 3926 ts_tablet_manager.cc:1918] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:10.789587 3926 log.cc:1196] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/435d025c029a487d8927656ddd6ba12e
I20260808 04:49:10.789906 3926 ts_tablet_manager.cc:1939] T 435d025c029a487d8927656ddd6ba12e P 553d5cb11c48493c87665011f6cebdee: Deleting consensus metadata
I20260808 04:49:10.791220 3093 catalog_manager.cc:4841] TS 553d5cb11c48493c87665011f6cebdee (127.4.217.2:41153): tablet 435d025c029a487d8927656ddd6ba12e (table test-table [id=f5e301dd25174f20bef89920222fb9cf]) successfully deleted
W20260808 04:49:10.792253 3091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:42177 (error 108)
I20260808 04:49:10.792537 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 3444
I20260808 04:49:10.795104 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:10.794713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:42177 (local address 127.0.0.1:50096)
0808 04:49:10.794892 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:10.794990 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 04:49:10.795285 3091 catalog_manager.cc:4584] TS 0ea7f0f44f874b1c80c7cac11debfccb (127.4.217.1:42177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 435d025c029a487d8927656ddd6ba12e: Network error: Client connection negotiation failed: client connection to 127.4.217.1:42177: connect: Connection refused (error 111)
I20260808 04:49:10.803576 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 3646
I20260808 04:49:10.815193 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 3069
2026-08-08T04:49:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1123 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:10 dist-test-slave-2kcd krb5kdc[3932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:10 dist-test-slave-2kcd krb5kdc[3932](info): set up 2 sockets
Aug 08 04:49:10 dist-test-slave-2kcd krb5kdc[3932](info): commencing operation
krb5kdc: starting...
W20260808 04:49:12.089625 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:12 dist-test-slave-2kcd krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:12.217890 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:38815
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:38815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:12.350593 3953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:12.350847 3953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:12.350886 3953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:12.354828 3953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:12.354882 3953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:12.354902 3953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:12.354923 3953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:12.354938 3953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:12.359989 3953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:38815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:38815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:12.360992 3953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:12.361961 3953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:12.370498 3959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:12.370661 3960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:12.370587 3953 server_base.cc:1034] running on GCE node
W20260808 04:49:12.370501 3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:12.371321 3953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:12.372449 3953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:12.373620 3953 hybrid_clock.cc:648] HybridClock initialized: now 1786164552373590 us; error 41 us; skew 500 ppm
Aug 08 04:49:12 dist-test-slave-2kcd krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164552, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:12.376365 3953 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:12.377351 3953 webserver.cc:460] Webserver started at http://127.4.217.62:33949/ using document root <none> and password file <none>
I20260808 04:49:12.377915 3953 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:12.378015 3953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:12.378257 3953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:12.380301 3953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "109ccca13f8c4426907f97b16ecafcf5"
format_stamp: "Formatted at 2026-08-08 04:49:12 on dist-test-slave-2kcd"
server_key: "ed209fafde3acd7102cca76e2c1a8a8f"
server_key_iv: "dad6f5665ad6f4318c6106f3247a411f"
server_key_version: "encryptionkey@0"
I20260808 04:49:12.380949 3953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "109ccca13f8c4426907f97b16ecafcf5"
format_stamp: "Formatted at 2026-08-08 04:49:12 on dist-test-slave-2kcd"
server_key: "ed209fafde3acd7102cca76e2c1a8a8f"
server_key_iv: "dad6f5665ad6f4318c6106f3247a411f"
server_key_version: "encryptionkey@0"
I20260808 04:49:12.384865 3953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:49:12.387269 3970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:12.388492 3953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:12.388646 3953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "109ccca13f8c4426907f97b16ecafcf5"
format_stamp: "Formatted at 2026-08-08 04:49:12 on dist-test-slave-2kcd"
server_key: "ed209fafde3acd7102cca76e2c1a8a8f"
server_key_iv: "dad6f5665ad6f4318c6106f3247a411f"
server_key_version: "encryptionkey@0"
I20260808 04:49:12.388763 3953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:12.410504 3953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:12.413352 3953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:12.413574 3953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:12.420943 3953 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:38815
I20260808 04:49:12.421064 4056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:38815 every 8 connection(s)
I20260808 04:49:12.421942 3953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:12.424564 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 3953
I20260808 04:49:12.424715 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:12.424892 4057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:12.425091 4964 external_mini_cluster.cc:1428] Setting key c70ab585f410e75b28e68d440630a0a5
I20260808 04:49:12.432081 4057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5: Bootstrap starting.
Aug 08 04:49:12 dist-test-slave-2kcd krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:12.434885 4057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:12.435875 4057 log.cc:824] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:12.438184 4057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5: No bootstrap required, opened a new log
I20260808 04:49:12.440167 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:12.426463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.4.217.62:38815)
0808 04:49:12.426886 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:12.426905 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:12.426952 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:49:12.427641 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:12.427659 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:12.427986 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:12.428308 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:12.429321 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.430316 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.431466 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.431906 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.432023 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:12.434794 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:12.434802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:12.434819 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:12.434856 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:12.436860 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:12.437261 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:12.437267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:12.437272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:12.437423 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:12.437680 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:12.437683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:12.437685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:12.438172 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:12.438374 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:12.438685 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 04:49:12.438970 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
I20260808 04:49:12.441992 4057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109ccca13f8c4426907f97b16ecafcf5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38815 } }
I20260808 04:49:12.442270 4057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:12.442376 4057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109ccca13f8c4426907f97b16ecafcf5, State: Initialized, Role: FOLLOWER
I20260808 04:49:12.442806 4057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109ccca13f8c4426907f97b16ecafcf5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38815 } }
I20260808 04:49:12.442950 4057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:12.443023 4057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:12.443140 4057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:12.443980 4057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109ccca13f8c4426907f97b16ecafcf5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38815 } }
I20260808 04:49:12.444321 4057 leader_election.cc:302] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 109ccca13f8c4426907f97b16ecafcf5; no voters:
I20260808 04:49:12.444614 4057 leader_election.cc:288] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:12.444741 4064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:12.444989 4064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [term 1 LEADER]: Becoming Leader. State: Replica: 109ccca13f8c4426907f97b16ecafcf5, State: Running, Role: LEADER
I20260808 04:49:12.445319 4064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109ccca13f8c4426907f97b16ecafcf5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38815 } }
I20260808 04:49:12.445796 4057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:12.447062 4069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 109ccca13f8c4426907f97b16ecafcf5. Latest consensus state: current_term: 1 leader_uuid: "109ccca13f8c4426907f97b16ecafcf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109ccca13f8c4426907f97b16ecafcf5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38815 } } }
I20260808 04:49:12.447063 4067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "109ccca13f8c4426907f97b16ecafcf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109ccca13f8c4426907f97b16ecafcf5" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 38815 } } }
I20260808 04:49:12.447229 4069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:12.447229 4067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:12.447804 4081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:12.451169 4081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:12.456739 4081 catalog_manager.cc:1347] Generated new cluster ID: 3ce4fc0c42a2433ea7d421534f8dddd4
I20260808 04:49:12.456842 4081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:12.479604 4081 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:12.480844 4081 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:12.486011 4081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5: Generated new TSK 0
I20260808 04:49:12.486661 4081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
Entry for principal kudu/127.4.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab.
I20260808 04:49:12.515681 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38815
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:12.630295 4094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:12.630565 4094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:12.630609 4094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:12.634639 4094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:12.634693 4094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:12.634752 4094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:49:12.639848 4094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.1.keytab
--principal=kudu/127.4.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:12.640894 4094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:12.641956 4094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:12.651975 4113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:12.651918 4109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:12.652206 4094 server_base.cc:1034] running on GCE node
W20260808 04:49:12.651926 4108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:12.652834 4094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:12.653475 4094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:12.654642 4094 hybrid_clock.cc:648] HybridClock initialized: now 1786164552654609 us; error 43 us; skew 500 ppm
Aug 08 04:49:12 dist-test-slave-2kcd krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164552, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:12.657389 4094 init.cc:377] Logged in from keytab as kudu/127.4.217.1@KRBTEST.COM (short username kudu)
I20260808 04:49:12.658221 4094 webserver.cc:460] Webserver started at http://127.4.217.1:44541/ using document root <none> and password file <none>
I20260808 04:49:12.658773 4094 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:12.658877 4094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:12.659106 4094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:12.661490 4094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "23484089b32249c89167de0f9ce0bb62"
format_stamp: "Formatted at 2026-08-08 04:49:12 on dist-test-slave-2kcd"
server_key: "0181aef82e464d28e4647406a671e84b"
server_key_iv: "f62e6a4ab086f1ef040ad8af75db02a7"
server_key_version: "encryptionkey@0"
I20260808 04:49:12.662055 4094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "23484089b32249c89167de0f9ce0bb62"
format_stamp: "Formatted at 2026-08-08 04:49:12 on dist-test-slave-2kcd"
server_key: "0181aef82e464d28e4647406a671e84b"
server_key_iv: "f62e6a4ab086f1ef040ad8af75db02a7"
server_key_version: "encryptionkey@0"
I20260808 04:49:12.665669 4094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:49:12.668009 4125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:12.669236 4094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:12.669368 4094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "23484089b32249c89167de0f9ce0bb62"
format_stamp: "Formatted at 2026-08-08 04:49:12 on dist-test-slave-2kcd"
server_key: "0181aef82e464d28e4647406a671e84b"
server_key_iv: "f62e6a4ab086f1ef040ad8af75db02a7"
server_key_version: "encryptionkey@0"
I20260808 04:49:12.669473 4094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:12.685585 4094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:12.688982 4094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:12.689188 4094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:12.689790 4094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:12.690623 4094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:12.690694 4094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:12.690768 4094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:12.690833 4094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:12.700145 4094 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:33741
I20260808 04:49:12.700208 4303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:33741 every 8 connection(s)
I20260808 04:49:12.701185 4094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:49:12.702092 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 4094
I20260808 04:49:12.702229 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:49:12.702543 4964 external_mini_cluster.cc:1428] Setting key 2bab84d2046c6702ce4e5e2c8c5bc261
Aug 08 04:49:12 dist-test-slave-2kcd krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164552, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.1@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.2@KRBTEST.COM" created.
I20260808 04:49:12.717365 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:12.703346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:49587 (local address 127.4.217.62:38815)
0808 04:49:12.703485 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:12.703488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:12.704611 (+ 1123us) server_negotiation.cc:407] Connection header received
0808 04:49:12.706004 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:12.706010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:12.706091 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:12.706189 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:12.707882 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.708411 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.710307 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.710534 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.710617 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:12.713563 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:12.713577 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:12.713579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:12.713606 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:12.715074 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:12.715641 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:12.715644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:12.715645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:12.715726 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:12.716043 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:12.716045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:12.716046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:12.716223 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:12.716298 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:12.716898 (+ 600us) server_negotiation.cc:299] Negotiation successful
0808 04:49:12.717104 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:49:12.718101 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:12.703638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38815 (local address 127.4.217.1:49587)
0808 04:49:12.704443 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:12.704498 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:12.705793 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:12.706431 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:12.706442 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:12.706969 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:12.707716 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:12.707741 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.708549 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.708556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:12.710178 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:12.710192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.710713 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.710721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:12.710966 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:12.711617 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:12.711646 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:12.713385 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:12.715208 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:12.715214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:12.715230 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:12.715523 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:12.715812 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:12.715814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:12.715815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:12.715949 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:12.716312 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:12.716316 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:12.716622 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 04:49:12.716815 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 04:49:12.717149 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":160,"threads_started":1}
I20260808 04:49:12.719383 4304 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38815
I20260808 04:49:12.719659 4304 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:12.720177 4304 heartbeater.cc:511] Master 127.4.217.62:38815 requested a full tablet report, sending...
I20260808 04:49:12.721990 3993 ts_manager.cc:194] Registered new tserver with Master: 23484089b32249c89167de0f9ce0bb62 (127.4.217.1:33741)
I20260808 04:49:12.723753 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.1@KRBTEST.COM'} at 127.4.217.1:49587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
Entry for principal kudu/127.4.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab.
I20260808 04:49:12.735692 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38815
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:12.845896 4310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:12.846129 4310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:12.846196 4310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:12.850633 4310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:12.850687 4310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:12.850742 4310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:49:12.855830 4310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.2.keytab
--principal=kudu/127.4.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:12.856765 4310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:12.857787 4310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:12.867158 4322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:12.867158 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:12.867203 4310 server_base.cc:1034] running on GCE node
W20260808 04:49:12.867190 4325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:12.867710 4310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:12.868314 4310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:12.869493 4310 hybrid_clock.cc:648] HybridClock initialized: now 1786164552869481 us; error 25 us; skew 500 ppm
Aug 08 04:49:12 dist-test-slave-2kcd krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164552, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:12.872138 4310 init.cc:377] Logged in from keytab as kudu/127.4.217.2@KRBTEST.COM (short username kudu)
I20260808 04:49:12.874644 4310 webserver.cc:460] Webserver started at http://127.4.217.2:40071/ using document root <none> and password file <none>
I20260808 04:49:12.875170 4310 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:12.875229 4310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:12.875463 4310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:12.877441 4310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94"
format_stamp: "Formatted at 2026-08-08 04:49:12 on dist-test-slave-2kcd"
server_key: "13827addee091d47445860eccc0bd578"
server_key_iv: "12b67be1fa824d1eb926592a5be9345f"
server_key_version: "encryptionkey@0"
I20260808 04:49:12.877972 4310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94"
format_stamp: "Formatted at 2026-08-08 04:49:12 on dist-test-slave-2kcd"
server_key: "13827addee091d47445860eccc0bd578"
server_key_iv: "12b67be1fa824d1eb926592a5be9345f"
server_key_version: "encryptionkey@0"
I20260808 04:49:12.881425 4310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:49:12.883634 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:12.884754 4310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:49:12.884938 4310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94"
format_stamp: "Formatted at 2026-08-08 04:49:12 on dist-test-slave-2kcd"
server_key: "13827addee091d47445860eccc0bd578"
server_key_iv: "12b67be1fa824d1eb926592a5be9345f"
server_key_version: "encryptionkey@0"
I20260808 04:49:12.885046 4310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:12.898924 4310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:12.901528 4310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:12.901729 4310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:12.902276 4310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:12.903029 4310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:12.903097 4310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:12.903167 4310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:12.903235 4310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:12.912081 4310 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:36363
I20260808 04:49:12.912109 4504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:36363 every 8 connection(s)
I20260808 04:49:12.912948 4310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
Aug 08 04:49:12 dist-test-slave-2kcd krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164552, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.2@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:12.922513 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 4310
I20260808 04:49:12.922698 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:49:12.923070 4964 external_mini_cluster.cc:1428] Setting key 39a850f7c423376d6e724ac6e621ff52
I20260808 04:49:12.926061 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:12.914624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:42093 (local address 127.4.217.62:38815)
0808 04:49:12.914774 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:12.914779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:12.915601 (+ 822us) server_negotiation.cc:407] Connection header received
0808 04:49:12.916998 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:12.917001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:12.917048 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:12.917122 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:12.918155 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.918674 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.919703 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.919913 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.919988 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:12.922278 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:12.922282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:12.922288 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:12.922324 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:12.923850 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:12.924417 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:12.924424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:12.924427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:12.924527 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:12.924898 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:12.924902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:12.924904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:12.925128 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:12.925241 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:12.925691 (+ 450us) server_negotiation.cc:299] Negotiation successful
0808 04:49:12.925828 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:49:12.926591 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:12.915014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38815 (local address 127.4.217.2:42093)
0808 04:49:12.915431 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:12.915470 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:12.916747 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:12.917269 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:12.917275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:12.917600 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:12.918030 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:12.918042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.918778 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.918780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:12.919597 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:12.919605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:12.920035 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:12.920038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:12.920165 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:12.920635 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:12.920650 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:12.922107 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:12.923958 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:12.923963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:12.923976 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:12.924302 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:12.924622 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:12.924626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:12.924627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:12.924767 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:12.925238 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:12.925243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:12.925441 (+ 198us) client_negotiation.cc:769] Sending connection context
0808 04:49:12.925587 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 04:49:12.925802 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":109,"threads_started":1}
I20260808 04:49:12.927749 4506 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38815
I20260808 04:49:12.927977 4506 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:12.928421 4506 heartbeater.cc:511] Master 127.4.217.62:38815 requested a full tablet report, sending...
I20260808 04:49:12.929466 3993 ts_manager.cc:194] Registered new tserver with Master: 6b496bbaf32a44f8b5b376fd2ffb7f94 (127.4.217.2:36363)
I20260808 04:49:12.930016 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.2@KRBTEST.COM'} at 127.4.217.2:42093
WARNING: no policy specified for kudu/127.4.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
Entry for principal kudu/127.4.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab.
I20260808 04:49:12.952255 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38815
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:13.066345 4511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:13.066584 4511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:13.066628 4511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:13.070623 4511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:13.070675 4511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:13.070734 4511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:49:13.076035 4511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.3.keytab
--principal=kudu/127.4.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:38815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:13.077033 4511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:13.077992 4511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:13.087627 4522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:13.087618 4526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:13.087628 4523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:13.087893 4511 server_base.cc:1034] running on GCE node
I20260808 04:49:13.088313 4511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:13.089032 4511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:13.090193 4511 hybrid_clock.cc:648] HybridClock initialized: now 1786164553090180 us; error 25 us; skew 500 ppm
Aug 08 04:49:13 dist-test-slave-2kcd krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164553, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:13.093104 4511 init.cc:377] Logged in from keytab as kudu/127.4.217.3@KRBTEST.COM (short username kudu)
I20260808 04:49:13.093850 4511 webserver.cc:460] Webserver started at http://127.4.217.3:40433/ using document root <none> and password file <none>
I20260808 04:49:13.094362 4511 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:13.094424 4511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:13.094583 4511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:13.096513 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "8337b45bd2ab4cd6a65391387556f82e"
format_stamp: "Formatted at 2026-08-08 04:49:13 on dist-test-slave-2kcd"
server_key: "5b2f6b5e225c988fcc00c5baf9aa83d7"
server_key_iv: "313f703a7d9e194163d0f3944f8473e1"
server_key_version: "encryptionkey@0"
I20260808 04:49:13.097067 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "8337b45bd2ab4cd6a65391387556f82e"
format_stamp: "Formatted at 2026-08-08 04:49:13 on dist-test-slave-2kcd"
server_key: "5b2f6b5e225c988fcc00c5baf9aa83d7"
server_key_iv: "313f703a7d9e194163d0f3944f8473e1"
server_key_version: "encryptionkey@0"
I20260808 04:49:13.100523 4511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:49:13.102847 4536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:13.104012 4511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:13.104143 4511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "8337b45bd2ab4cd6a65391387556f82e"
format_stamp: "Formatted at 2026-08-08 04:49:13 on dist-test-slave-2kcd"
server_key: "5b2f6b5e225c988fcc00c5baf9aa83d7"
server_key_iv: "313f703a7d9e194163d0f3944f8473e1"
server_key_version: "encryptionkey@0"
I20260808 04:49:13.104249 4511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:13.126860 4511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:13.129844 4511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:13.130060 4511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:13.130663 4511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:13.131515 4511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:13.131588 4511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:13.131661 4511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:13.131704 4511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:13.140658 4511 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:43077
I20260808 04:49:13.140693 4712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:43077 every 8 connection(s)
I20260808 04:49:13.141539 4511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:49:13.148706 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 4511
I20260808 04:49:13.148900 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:49:13.149276 4964 external_mini_cluster.cc:1428] Setting key 710541740876b2a5e62aef90d380a9fd
Aug 08 04:49:13 dist-test-slave-2kcd krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164553, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.3@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:13.155529 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.143287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:45253 (local address 127.4.217.62:38815)
0808 04:49:13.143427 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.143431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.144229 (+ 798us) server_negotiation.cc:407] Connection header received
0808 04:49:13.145742 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.145745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.145804 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.145907 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:13.147222 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.147834 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.149084 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.149426 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.149548 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.152318 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:13.152321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:13.152323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:13.152348 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:13.153706 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:13.154184 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:13.154187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:13.154188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:13.154269 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:13.154581 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:13.154583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:13.154584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:13.154761 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:13.154838 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.155303 (+ 465us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.155397 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:49:13.156368 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.143634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:38815 (local address 127.4.217.3:45253)
0808 04:49:13.144086 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:13.144128 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:13.145528 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:13.146100 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:13.146107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:13.146608 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:13.147075 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.147089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.147962 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.147966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.148933 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.148949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.149577 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.149580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.149778 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.150425 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:13.150446 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:13.152169 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:13.153811 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:13.153816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:13.153827 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:13.154095 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:13.154353 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:13.154355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:13.154356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:13.154499 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:13.154853 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:13.154858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:13.155097 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 04:49:13.155254 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 04:49:13.155501 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260808 04:49:13.157578 4715 heartbeater.cc:347] Connected to a master server at 127.4.217.62:38815
I20260808 04:49:13.157794 4715 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:13.158241 4715 heartbeater.cc:511] Master 127.4.217.62:38815 requested a full tablet report, sending...
I20260808 04:49:13.159237 3993 ts_manager.cc:194] Registered new tserver with Master: 8337b45bd2ab4cd6a65391387556f82e (127.4.217.3:43077)
I20260808 04:49:13.159881 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.217.3@KRBTEST.COM'} at 127.4.217.3:45253
I20260808 04:49:13.163277 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:49:13.172538 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.165328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.4.217.62:38815)
0808 04:49:13.165495 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.165500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.165640 (+ 140us) server_negotiation.cc:407] Connection header received
0808 04:49:13.165786 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.165790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.165831 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.165929 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:13.166618 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.167142 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.167997 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.168235 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.168345 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.169667 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:13.169671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:13.169673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:13.169698 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:13.170832 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:13.171284 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:13.171288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:13.171290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:13.171392 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:13.171662 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:13.171666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:13.171668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:13.171896 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:13.172040 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.172285 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.172395 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:49:13.175406 3993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:49:13.178406 3993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:49:13.192699 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.186574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:43077 (local address 127.0.0.1:52484)
0808 04:49:13.187442 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:13.187490 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:13.187618 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:13.188323 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:13.188328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:13.188350 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:13.188504 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.188515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.189751 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.189757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.190961 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.190971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.192128 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.192131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.192270 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.192308 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:49:13.192430 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 04:49:13.192543 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":5,"thread_start_us":76,"threads_started":1}
I20260808 04:49:13.193092 4737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.186720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.4.217.3:43077)
0808 04:49:13.187116 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.187128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.187756 (+ 628us) server_negotiation.cc:407] Connection header received
0808 04:49:13.187840 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.187845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.188112 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.188277 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:13.188660 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.189619 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.191090 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.191795 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.191903 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.192064 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.192487 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.192587 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":162,"threads_started":1}
I20260808 04:49:13.193545 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.187329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.4.217.1:33741)
0808 04:49:13.187712 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.187730 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.188050 (+ 320us) server_negotiation.cc:407] Connection header received
0808 04:49:13.188329 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.188335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.188535 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.188717 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:13.189184 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.190290 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.191895 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.192598 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.192703 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.192881 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.193281 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.193382 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":93,"threads_started":1}
I20260808 04:49:13.193581 4739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.187192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:36363 (local address 127.0.0.1:60754)
0808 04:49:13.188026 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:13.188040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:13.188133 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:13.188631 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:13.188636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:13.188660 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:13.188761 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.188770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.190323 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.190327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.191663 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.191675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.193184 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.193188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.193278 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.193302 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:49:13.193372 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 04:49:13.193445 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":33,"spinlock_wait_cycles":2304,"thread_start_us":94,"threads_started":1}
I20260808 04:49:13.193972 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.186751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33741 (local address 127.0.0.1:45786)
0808 04:49:13.187948 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:13.187972 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:13.188141 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:13.188862 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:13.188865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:13.188884 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:13.188991 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.188998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.190430 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.190433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.191747 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.191758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.192943 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.192947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.193096 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.193131 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:49:13.193753 (+ 622us) client_negotiation.cc:240] Negotiation successful
0808 04:49:13.193827 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":67,"spinlock_wait_cycles":10496,"thread_start_us":95,"threads_started":1}
I20260808 04:49:13.194029 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.187622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.4.217.2:36363)
0808 04:49:13.188095 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.188107 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.188127 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:49:13.188249 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.188254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.188461 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.188643 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:13.188931 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.190168 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.191818 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.192730 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.192911 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.193100 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.193796 (+ 696us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.193897 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":155,"threads_started":1}
I20260808 04:49:13.196058 4608 tablet_service.cc:1467] Processing CreateTablet for tablet a5777704e41c436aa2c5d10bf7abcc3b (DEFAULT_TABLE table=test-table [id=b5b9879c40f043508e2927a60cec9b3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:13.197095 4608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5777704e41c436aa2c5d10bf7abcc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:13.196660 4402 tablet_service.cc:1467] Processing CreateTablet for tablet a5777704e41c436aa2c5d10bf7abcc3b (DEFAULT_TABLE table=test-table [id=b5b9879c40f043508e2927a60cec9b3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:13.196663 4193 tablet_service.cc:1467] Processing CreateTablet for tablet a5777704e41c436aa2c5d10bf7abcc3b (DEFAULT_TABLE table=test-table [id=b5b9879c40f043508e2927a60cec9b3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:13.197664 4402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5777704e41c436aa2c5d10bf7abcc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:13.197664 4193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5777704e41c436aa2c5d10bf7abcc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:13.203227 4743 tablet_bootstrap.cc:492] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: Bootstrap starting.
I20260808 04:49:13.204337 4744 tablet_bootstrap.cc:492] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62: Bootstrap starting.
I20260808 04:49:13.205519 4748 tablet_bootstrap.cc:492] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: Bootstrap starting.
I20260808 04:49:13.205614 4743 tablet_bootstrap.cc:654] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:13.206470 4744 tablet_bootstrap.cc:654] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:13.207021 4743 log.cc:824] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:13.207780 4748 tablet_bootstrap.cc:654] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:13.207851 4744 log.cc:824] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:13.208636 4748 log.cc:824] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:13.209777 4744 tablet_bootstrap.cc:492] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62: No bootstrap required, opened a new log
I20260808 04:49:13.209782 4743 tablet_bootstrap.cc:492] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: No bootstrap required, opened a new log
I20260808 04:49:13.209965 4743 ts_tablet_manager.cc:1397] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:49:13.209965 4744 ts_tablet_manager.cc:1397] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:49:13.210311 4748 tablet_bootstrap.cc:492] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: No bootstrap required, opened a new log
I20260808 04:49:13.210510 4748 ts_tablet_manager.cc:1397] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:49:13.213249 4748 raft_consensus.cc:348] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } }
I20260808 04:49:13.213498 4748 raft_consensus.cc:374] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:13.213554 4748 raft_consensus.cc:729] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b496bbaf32a44f8b5b376fd2ffb7f94, State: Initialized, Role: FOLLOWER
I20260808 04:49:13.213820 4744 raft_consensus.cc:348] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } }
I20260808 04:49:13.213820 4743 raft_consensus.cc:348] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } }
I20260808 04:49:13.214075 4744 raft_consensus.cc:374] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:13.214075 4743 raft_consensus.cc:374] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:13.213995 4748 consensus_queue.cc:260] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } }
I20260808 04:49:13.214135 4743 raft_consensus.cc:729] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8337b45bd2ab4cd6a65391387556f82e, State: Initialized, Role: FOLLOWER
I20260808 04:49:13.214135 4744 raft_consensus.cc:729] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23484089b32249c89167de0f9ce0bb62, State: Initialized, Role: FOLLOWER
I20260808 04:49:13.214587 4744 consensus_queue.cc:260] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } }
I20260808 04:49:13.214587 4743 consensus_queue.cc:260] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } }
I20260808 04:49:13.215063 4748 ts_tablet_manager.cc:1428] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260808 04:49:13.215404 4506 heartbeater.cc:503] Master 127.4.217.62:38815 was elected leader, sending a full tablet report...
I20260808 04:49:13.215482 4744 ts_tablet_manager.cc:1428] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 04:49:13.215823 4304 heartbeater.cc:503] Master 127.4.217.62:38815 was elected leader, sending a full tablet report...
I20260808 04:49:13.215883 4743 ts_tablet_manager.cc:1428] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:49:13.216288 4715 heartbeater.cc:503] Master 127.4.217.62:38815 was elected leader, sending a full tablet report...
I20260808 04:49:13.321522 4754 raft_consensus.cc:482] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:49:13.321808 4754 raft_consensus.cc:504] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } }
I20260808 04:49:13.322991 4754 leader_election.cc:288] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8337b45bd2ab4cd6a65391387556f82e (127.4.217.3:43077), 6b496bbaf32a44f8b5b376fd2ffb7f94 (127.4.217.2:36363)
I20260808 04:49:13.326819 4737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.323374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:44895 (local address 127.4.217.3:43077)
0808 04:49:13.323516 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.323521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.323632 (+ 111us) server_negotiation.cc:407] Connection header received
0808 04:49:13.323802 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.323805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.323854 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.323945 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:13.324235 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.324742 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.325845 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.326259 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.326379 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.326458 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.326630 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.326700 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:49:13.326835 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.323240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:43077 (local address 127.4.217.1:44895)
0808 04:49:13.323529 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:13.323543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:13.323676 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:13.323962 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:13.323965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:13.323985 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:13.324118 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.324124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.324910 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.324914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.325714 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.325723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.326385 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.326389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.326490 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.326514 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:49:13.326611 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 04:49:13.326680 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260808 04:49:13.327289 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.323418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:33113 (local address 127.4.217.2:36363)
0808 04:49:13.323586 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.323592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.324044 (+ 452us) server_negotiation.cc:407] Connection header received
0808 04:49:13.324196 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.324199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.324251 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.324338 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:13.324604 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.325148 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.326158 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.326588 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.326703 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.326764 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.327108 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.327197 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:49:13.327536 4648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5777704e41c436aa2c5d10bf7abcc3b" candidate_uuid: "23484089b32249c89167de0f9ce0bb62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8337b45bd2ab4cd6a65391387556f82e" is_pre_election: true
I20260808 04:49:13.327684 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.323240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:36363 (local address 127.4.217.1:33113)
0808 04:49:13.323933 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:13.323944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:13.324075 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:13.324348 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:13.324351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:13.324366 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:13.324485 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.324491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.325257 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.325262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.326029 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.326039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.326710 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.326714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.326819 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.326846 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:49:13.327456 (+ 610us) client_negotiation.cc:240] Negotiation successful
0808 04:49:13.327541 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":292,"thread_start_us":187,"threads_started":1}
I20260808 04:49:13.327842 4648 raft_consensus.cc:2454] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23484089b32249c89167de0f9ce0bb62 in term 0.
I20260808 04:49:13.328367 4134 leader_election.cc:302] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23484089b32249c89167de0f9ce0bb62, 8337b45bd2ab4cd6a65391387556f82e; no voters:
I20260808 04:49:13.328368 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5777704e41c436aa2c5d10bf7abcc3b" candidate_uuid: "23484089b32249c89167de0f9ce0bb62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" is_pre_election: true
I20260808 04:49:13.328675 4432 raft_consensus.cc:2454] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23484089b32249c89167de0f9ce0bb62 in term 0.
I20260808 04:49:13.328698 4754 raft_consensus.cc:2781] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:49:13.328812 4754 raft_consensus.cc:482] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:49:13.328877 4754 raft_consensus.cc:3037] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:13.329957 4754 raft_consensus.cc:504] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } }
I20260808 04:49:13.330300 4754 leader_election.cc:288] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [CANDIDATE]: Term 1 election: Requested vote from peers 8337b45bd2ab4cd6a65391387556f82e (127.4.217.3:43077), 6b496bbaf32a44f8b5b376fd2ffb7f94 (127.4.217.2:36363)
I20260808 04:49:13.330672 4648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5777704e41c436aa2c5d10bf7abcc3b" candidate_uuid: "23484089b32249c89167de0f9ce0bb62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8337b45bd2ab4cd6a65391387556f82e"
I20260808 04:49:13.330694 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5777704e41c436aa2c5d10bf7abcc3b" candidate_uuid: "23484089b32249c89167de0f9ce0bb62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94"
I20260808 04:49:13.330802 4648 raft_consensus.cc:3037] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:13.330801 4432 raft_consensus.cc:3037] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:13.332080 4432 raft_consensus.cc:2454] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23484089b32249c89167de0f9ce0bb62 in term 1.
I20260808 04:49:13.332080 4648 raft_consensus.cc:2454] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23484089b32249c89167de0f9ce0bb62 in term 1.
I20260808 04:49:13.332419 4128 leader_election.cc:302] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23484089b32249c89167de0f9ce0bb62, 6b496bbaf32a44f8b5b376fd2ffb7f94; no voters:
I20260808 04:49:13.332580 4754 raft_consensus.cc:2781] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:13.332823 4754 raft_consensus.cc:686] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 1 LEADER]: Becoming Leader. State: Replica: 23484089b32249c89167de0f9ce0bb62, State: Running, Role: LEADER
I20260808 04:49:13.333149 4754 consensus_queue.cc:237] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } }
I20260808 04:49:13.336372 3991 catalog_manager.cc:5495] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23484089b32249c89167de0f9ce0bb62 (127.4.217.1). New cstate: current_term: 1 leader_uuid: "23484089b32249c89167de0f9ce0bb62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23484089b32249c89167de0f9ce0bb62" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 33741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260808 04:49:13.381008 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.377304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.4.217.1:33741)
0808 04:49:13.377462 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.377468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.377485 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:49:13.377584 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.377587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.377644 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.377760 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:13.378038 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.378598 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.379479 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.379701 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.379809 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.380150 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:13.380571 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:13.380710 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.380797 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.380858 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:49:13.387311 4432 raft_consensus.cc:1261] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [term 1 FOLLOWER]: Refusing update from remote peer 23484089b32249c89167de0f9ce0bb62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:13.387311 4648 raft_consensus.cc:1261] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [term 1 FOLLOWER]: Refusing update from remote peer 23484089b32249c89167de0f9ce0bb62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:13.388077 4754 consensus_queue.cc:1035] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b496bbaf32a44f8b5b376fd2ffb7f94" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:13.388285 4761 consensus_queue.cc:1035] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8337b45bd2ab4cd6a65391387556f82e" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:49:13.392419 4716 tablet.cc:2369] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:49:13.396592 4764 mvcc.cc:205] Tried to move back new op lower bound from 7316130010667061248 to 7316130010455232512. Current Snapshot: MvccSnapshot[applied={T|T < 7316130010667061248}]
I20260808 04:49:13.404743 3993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:13.404980 3993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:13.407361 3993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 109ccca13f8c4426907f97b16ecafcf5: Sending DeleteTablet for 3 replicas of tablet a5777704e41c436aa2c5d10bf7abcc3b
I20260808 04:49:13.408177 4608 tablet_service.cc:1514] Processing DeleteTablet for tablet a5777704e41c436aa2c5d10bf7abcc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:13 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:52484
I20260808 04:49:13.408213 4193 tablet_service.cc:1514] Processing DeleteTablet for tablet a5777704e41c436aa2c5d10bf7abcc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:13 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:45786
I20260808 04:49:13.408324 4402 tablet_service.cc:1514] Processing DeleteTablet for tablet a5777704e41c436aa2c5d10bf7abcc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:13 UTC) from {username='kudu', principal='kudu/127.4.217.62@KRBTEST.COM'} at 127.0.0.1:60754
I20260808 04:49:13.408871 4782 tablet_replica.cc:330] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62: stopping tablet replica
I20260808 04:49:13.409080 4783 tablet_replica.cc:330] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: stopping tablet replica
I20260808 04:49:13.409186 4782 raft_consensus.cc:2229] T a5777704e41c436aa2c5d10bf7abcc3b P 23484089b32249c89167de0f9ce0bb62 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:49:13.409479 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 4094
I20260808 04:49:13.413034 4783 raft_consensus.cc:2229] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:13.413497 4783 raft_consensus.cc:2258] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:13.419610 4783 ts_tablet_manager.cc:1905] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:13.428634 4783 ts_tablet_manager.cc:1918] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:13.428768 4783 log.cc:1196] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/a5777704e41c436aa2c5d10bf7abcc3b
I20260808 04:49:13.429105 4783 ts_tablet_manager.cc:1939] T a5777704e41c436aa2c5d10bf7abcc3b P 6b496bbaf32a44f8b5b376fd2ffb7f94: Deleting consensus metadata
I20260808 04:49:13.430429 4779 tablet_replica.cc:330] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: stopping tablet replica
I20260808 04:49:13.430392 3972 catalog_manager.cc:4841] TS 6b496bbaf32a44f8b5b376fd2ffb7f94 (127.4.217.2:36363): tablet a5777704e41c436aa2c5d10bf7abcc3b (table test-table [id=b5b9879c40f043508e2927a60cec9b3a]) successfully deleted
I20260808 04:49:13.430805 4779 raft_consensus.cc:2229] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:13.431143 4779 raft_consensus.cc:2258] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:13.432583 4779 ts_tablet_manager.cc:1905] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:13.434945 4779 ts_tablet_manager.cc:1918] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260808 04:49:13.434945 3971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:33741 (error 108)
I20260808 04:49:13.435060 4779 log.cc:1196] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/a5777704e41c436aa2c5d10bf7abcc3b
I20260808 04:49:13.435454 4779 ts_tablet_manager.cc:1939] T a5777704e41c436aa2c5d10bf7abcc3b P 8337b45bd2ab4cd6a65391387556f82e: Deleting consensus metadata
I20260808 04:49:13.435832 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 4310
I20260808 04:49:13.436160 4739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.435822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:33741 (local address 127.0.0.1:45792)
0808 04:49:13.436007 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:13.436075 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:33741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260808 04:49:13.436426 3971 catalog_manager.cc:4584] TS 23484089b32249c89167de0f9ce0bb62 (127.4.217.1:33741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5777704e41c436aa2c5d10bf7abcc3b: Network error: Client connection negotiation failed: client connection to 127.4.217.1:33741: connect: Connection refused (error 111)
I20260808 04:49:13.438381 3974 catalog_manager.cc:4841] TS 8337b45bd2ab4cd6a65391387556f82e (127.4.217.3:43077): tablet a5777704e41c436aa2c5d10bf7abcc3b (table test-table [id=b5b9879c40f043508e2927a60cec9b3a]) successfully deleted
I20260808 04:49:13.446645 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 4511
I20260808 04:49:13.461939 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 3953
2026-08-08T04:49:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2641 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T04:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:13Z Disabled control of system clock
I20260808 04:49:13.502146 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41739
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:49:13.615348 4790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:13.615574 4790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:13.615628 4790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:13.619467 4790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:13.619516 4790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:13.619532 4790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:13.619549 4790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:13.619563 4790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:13.624116 4790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:13.625052 4790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:13.626300 4790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:13.634603 4797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:13.634608 4798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:13.634672 4800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:13.634806 4790 server_base.cc:1034] running on GCE node
I20260808 04:49:13.635411 4790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:13.636442 4790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:13.637600 4790 hybrid_clock.cc:648] HybridClock initialized: now 1786164553637564 us; error 51 us; skew 500 ppm
I20260808 04:49:13.639389 4790 webserver.cc:460] Webserver started at http://127.4.217.62:43243/ using document root <none> and password file <none>
I20260808 04:49:13.639950 4790 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:13.640050 4790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:13.640280 4790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:13.642294 4790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "8ddc9e83ef3845148bdb4c28a7a2152d"
format_stamp: "Formatted at 2026-08-08 04:49:13 on dist-test-slave-2kcd"
server_key: "6bc1950e7f0e622627a2c734e85d2614"
server_key_iv: "1653931fb9a9dfb3a2781b5414bf823f"
server_key_version: "encryptionkey@0"
I20260808 04:49:13.642845 4790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "8ddc9e83ef3845148bdb4c28a7a2152d"
format_stamp: "Formatted at 2026-08-08 04:49:13 on dist-test-slave-2kcd"
server_key: "6bc1950e7f0e622627a2c734e85d2614"
server_key_iv: "1653931fb9a9dfb3a2781b5414bf823f"
server_key_version: "encryptionkey@0"
I20260808 04:49:13.646348 4790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:49:13.648604 4808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:13.649760 4790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:49:13.649900 4790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "8ddc9e83ef3845148bdb4c28a7a2152d"
format_stamp: "Formatted at 2026-08-08 04:49:13 on dist-test-slave-2kcd"
server_key: "6bc1950e7f0e622627a2c734e85d2614"
server_key_iv: "1653931fb9a9dfb3a2781b5414bf823f"
server_key_version: "encryptionkey@0"
I20260808 04:49:13.650008 4790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:13.667105 4790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:13.667711 4790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:13.667888 4790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:13.675048 4790 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:41739
I20260808 04:49:13.675096 4881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:41739 every 8 connection(s)
I20260808 04:49:13.676065 4790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:13.677706 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 4790
I20260808 04:49:13.677845 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:13.678171 4964 external_mini_cluster.cc:1428] Setting key 41ebbf245524480c0d88ed1ec2770c3e
I20260808 04:49:13.679618 4882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:13.685964 4882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d: Bootstrap starting.
I20260808 04:49:13.687326 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.679588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.4.217.62:41739)
0808 04:49:13.680232 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.680247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.680283 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:49:13.681070 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.681088 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.681446 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.681720 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:13.682067 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.683000 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.684179 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.684627 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.684756 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.685743 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:13.685748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:13.685759 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:13.686186 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:13.686235 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:13.686254 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:13.686359 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.686473 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.686708 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":144,"threads_started":1}
I20260808 04:49:13.688163 4882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:13.689100 4882 log.cc:824] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:13.690917 4882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d: No bootstrap required, opened a new log
I20260808 04:49:13.694334 4882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddc9e83ef3845148bdb4c28a7a2152d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41739 } }
I20260808 04:49:13.694538 4882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:13.694574 4882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddc9e83ef3845148bdb4c28a7a2152d, State: Initialized, Role: FOLLOWER
I20260808 04:49:13.694983 4882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddc9e83ef3845148bdb4c28a7a2152d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41739 } }
I20260808 04:49:13.695147 4882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:13.695219 4882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:13.695367 4882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:13.696305 4882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddc9e83ef3845148bdb4c28a7a2152d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41739 } }
I20260808 04:49:13.696589 4882 leader_election.cc:302] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ddc9e83ef3845148bdb4c28a7a2152d; no voters:
I20260808 04:49:13.696878 4882 leader_election.cc:288] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:13.696950 4892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:13.697292 4892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [term 1 LEADER]: Becoming Leader. State: Replica: 8ddc9e83ef3845148bdb4c28a7a2152d, State: Running, Role: LEADER
I20260808 04:49:13.697567 4892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddc9e83ef3845148bdb4c28a7a2152d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41739 } }
I20260808 04:49:13.698089 4882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:13.699294 4895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ddc9e83ef3845148bdb4c28a7a2152d. Latest consensus state: current_term: 1 leader_uuid: "8ddc9e83ef3845148bdb4c28a7a2152d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddc9e83ef3845148bdb4c28a7a2152d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41739 } } }
I20260808 04:49:13.699416 4895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:13.699674 4893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ddc9e83ef3845148bdb4c28a7a2152d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddc9e83ef3845148bdb4c28a7a2152d" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41739 } } }
I20260808 04:49:13.700285 4904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:13.699779 4893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:13.703297 4904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:13.708428 4904 catalog_manager.cc:1347] Generated new cluster ID: e100fc1c84dd4d80b4bb9350f1438de2
I20260808 04:49:13.708483 4904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:13.717120 4904 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:13.718184 4904 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:13.725863 4904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d: Generated new TSK 0
I20260808 04:49:13.726488 4904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:13.729566 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:41739
--builtin_ntp_servers=127.4.217.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:49:13.842969 4921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:13.843225 4921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:13.843264 4921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:13.847435 4921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:13.847489 4921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:13.847566 4921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20260808 04:49:13.852305 4921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:13.853444 4921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:13.854837 4921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:13.864725 4939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:13.864739 4932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:13.864888 4921 server_base.cc:1034] running on GCE node
W20260808 04:49:13.865101 4931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:13.865571 4921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:13.866230 4921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:13.867429 4921 hybrid_clock.cc:648] HybridClock initialized: now 1786164553867392 us; error 48 us; skew 500 ppm
I20260808 04:49:13.872982 4921 webserver.cc:460] Webserver started at http://127.4.217.1:36207/ using document root <none> and password file <none>
I20260808 04:49:13.873603 4921 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:13.873701 4921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:13.873950 4921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:13.876202 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/data/instance:
uuid: "5ab3a2c00b884e748f99c48f05aa8238"
format_stamp: "Formatted at 2026-08-08 04:49:13 on dist-test-slave-2kcd"
server_key: "b34e67603108bcc14d597fd8d1acd539"
server_key_iv: "aa3a7c7ea6c5043f74ba13e36dcf4675"
server_key_version: "encryptionkey@0"
I20260808 04:49:13.876866 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "5ab3a2c00b884e748f99c48f05aa8238"
format_stamp: "Formatted at 2026-08-08 04:49:13 on dist-test-slave-2kcd"
server_key: "b34e67603108bcc14d597fd8d1acd539"
server_key_iv: "aa3a7c7ea6c5043f74ba13e36dcf4675"
server_key_version: "encryptionkey@0"
I20260808 04:49:13.880635 4921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:49:13.883231 4945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:13.884469 4921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:13.884610 4921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/wal
uuid: "5ab3a2c00b884e748f99c48f05aa8238"
format_stamp: "Formatted at 2026-08-08 04:49:13 on dist-test-slave-2kcd"
server_key: "b34e67603108bcc14d597fd8d1acd539"
server_key_iv: "aa3a7c7ea6c5043f74ba13e36dcf4675"
server_key_version: "encryptionkey@0"
I20260808 04:49:13.884730 4921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:13.917502 4921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:13.918282 4921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:13.918505 4921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:13.919181 4921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:13.920388 4921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:13.920451 4921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:13.920492 4921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:13.920522 4921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:13.930385 4921 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:38775
I20260808 04:49:13.930454 5129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.1:38775 every 8 connection(s)
I20260808 04:49:13.931317 4921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/data/info.pb
I20260808 04:49:13.937335 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 4921
I20260808 04:49:13.937486 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-0/wal/instance
I20260808 04:49:13.937796 4964 external_mini_cluster.cc:1428] Setting key 99644d4a1b2296eb677355f2fb86ff13
I20260808 04:49:13.939909 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:41739
--builtin_ntp_servers=127.4.217.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:49:13.941282 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.933260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.1:58943 (local address 127.4.217.62:41739)
0808 04:49:13.933438 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:13.933443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:13.934512 (+ 1069us) server_negotiation.cc:407] Connection header received
0808 04:49:13.935939 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:13.935942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:13.935998 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:13.936080 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:13.937066 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.937697 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.939036 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.939343 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.939454 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.940609 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:13.940613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:13.940615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:13.940629 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:13.940651 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:13.940665 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:13.940740 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:13.940971 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 04:49:13.941071 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:49:13.942109 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:13.933689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41739 (local address 127.4.217.1:58943)
0808 04:49:13.934351 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:13.934391 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:13.935690 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:13.936243 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:13.936249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:13.936332 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:13.936906 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.936923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.937828 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.937834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.938899 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:13.938910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:13.939494 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:13.939498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:13.939682 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:13.940317 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:49:13.940318 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:13.940360 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:49:13.940442 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:13.940755 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:13.940768 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:13.940818 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 04:49:13.940968 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 04:49:13.941274 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260808 04:49:13.943588 5130 heartbeater.cc:347] Connected to a master server at 127.4.217.62:41739
I20260808 04:49:13.943831 5130 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:13.944356 5130 heartbeater.cc:511] Master 127.4.217.62:41739 requested a full tablet report, sending...
I20260808 04:49:13.946215 4835 ts_manager.cc:194] Registered new tserver with Master: 5ab3a2c00b884e748f99c48f05aa8238 (127.4.217.1:38775)
I20260808 04:49:13.948001 4835 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.1:58943
W20260808 04:49:14.058564 5135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:14.058812 5135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:14.058849 5135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:14.062968 5135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:14.063023 5135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:14.063074 5135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20260808 04:49:14.067741 5135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:14.068708 5135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:14.070072 5135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:14.079706 5145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:14.079877 5135 server_base.cc:1034] running on GCE node
W20260808 04:49:14.079747 5148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:14.079723 5146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:14.080386 5135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:14.081111 5135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:14.082310 5135 hybrid_clock.cc:648] HybridClock initialized: now 1786164554082284 us; error 34 us; skew 500 ppm
I20260808 04:49:14.084213 5135 webserver.cc:460] Webserver started at http://127.4.217.2:43843/ using document root <none> and password file <none>
I20260808 04:49:14.084858 5135 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:14.084952 5135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:14.085177 5135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:14.087167 5135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/data/instance:
uuid: "600e69544b254e23977f7d38d03c4013"
format_stamp: "Formatted at 2026-08-08 04:49:14 on dist-test-slave-2kcd"
server_key: "52a035c203d4849198a645caa26af447"
server_key_iv: "b9871e43354c9689ca4e2f512a2f821b"
server_key_version: "encryptionkey@0"
I20260808 04:49:14.087831 5135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "600e69544b254e23977f7d38d03c4013"
format_stamp: "Formatted at 2026-08-08 04:49:14 on dist-test-slave-2kcd"
server_key: "52a035c203d4849198a645caa26af447"
server_key_iv: "b9871e43354c9689ca4e2f512a2f821b"
server_key_version: "encryptionkey@0"
I20260808 04:49:14.091485 5135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 04:49:14.093815 5158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:14.094952 5135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:14.095100 5135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/wal
uuid: "600e69544b254e23977f7d38d03c4013"
format_stamp: "Formatted at 2026-08-08 04:49:14 on dist-test-slave-2kcd"
server_key: "52a035c203d4849198a645caa26af447"
server_key_iv: "b9871e43354c9689ca4e2f512a2f821b"
server_key_version: "encryptionkey@0"
I20260808 04:49:14.095208 5135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:14.118624 5135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:14.119225 5135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:14.119423 5135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:14.119976 5135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:14.120893 5135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:14.120965 5135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:14.121033 5135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:14.121084 5135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:14.130028 5135 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:44583
I20260808 04:49:14.130069 5347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.2:44583 every 8 connection(s)
I20260808 04:49:14.130856 5135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/data/info.pb
I20260808 04:49:14.138134 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 5135
I20260808 04:49:14.138290 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/wal/instance
I20260808 04:49:14.138604 4964 external_mini_cluster.cc:1428] Setting key 788a1fe829feaebbb28c6fe08840de6d
I20260808 04:49:14.139815 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.132568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:56127 (local address 127.4.217.62:41739)
0808 04:49:14.132704 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:14.132708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:14.133755 (+ 1047us) server_negotiation.cc:407] Connection header received
0808 04:49:14.135137 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:14.135140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:14.135202 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:14.135312 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:14.136141 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.136666 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.137819 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.138056 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.138170 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.139319 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:14.139322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:14.139324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:14.139336 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:14.139353 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:14.139361 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:14.139441 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:14.139597 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 04:49:14.139661 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:49:14.140825 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.132998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41739 (local address 127.4.217.2:56127)
0808 04:49:14.133609 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:14.133655 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:14.134916 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:14.135483 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:14.135490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:14.135568 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:14.135994 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.136005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.136816 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.136820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.137683 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.137693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.138212 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.138216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.138412 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.139058 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:49:14.139060 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:14.139099 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:49:14.139157 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:14.139443 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:14.139456 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:14.139464 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:49:14.139596 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 04:49:14.139850 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260808 04:49:14.140865 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:41739
--builtin_ntp_servers=127.4.217.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:49:14.142050 5348 heartbeater.cc:347] Connected to a master server at 127.4.217.62:41739
I20260808 04:49:14.142287 5348 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:14.142879 5348 heartbeater.cc:511] Master 127.4.217.62:41739 requested a full tablet report, sending...
I20260808 04:49:14.143970 4835 ts_manager.cc:194] Registered new tserver with Master: 600e69544b254e23977f7d38d03c4013 (127.4.217.2:44583)
I20260808 04:49:14.144527 4835 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.2:56127
W20260808 04:49:14.259670 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:14.259976 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:14.260016 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:14.264361 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:14.264443 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:14.264515 5353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20260808 04:49:14.269629 5353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:14.270630 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:14.271962 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:14.281719 5369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:14.281728 5365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:14.281778 5367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:14.282079 5353 server_base.cc:1034] running on GCE node
I20260808 04:49:14.282473 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:14.283090 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:14.284252 5353 hybrid_clock.cc:648] HybridClock initialized: now 1786164554284236 us; error 27 us; skew 500 ppm
I20260808 04:49:14.286010 5353 webserver.cc:460] Webserver started at http://127.4.217.3:44567/ using document root <none> and password file <none>
I20260808 04:49:14.286569 5353 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:14.286628 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:14.286784 5353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:14.288993 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/data/instance:
uuid: "e6a413b32eeb49918801e9d08367d4e3"
format_stamp: "Formatted at 2026-08-08 04:49:14 on dist-test-slave-2kcd"
server_key: "a13017d752dab105fab79fdda2631c5d"
server_key_iv: "6094e8caece5722db0ea54d81e482e88"
server_key_version: "encryptionkey@0"
I20260808 04:49:14.289682 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "e6a413b32eeb49918801e9d08367d4e3"
format_stamp: "Formatted at 2026-08-08 04:49:14 on dist-test-slave-2kcd"
server_key: "a13017d752dab105fab79fdda2631c5d"
server_key_iv: "6094e8caece5722db0ea54d81e482e88"
server_key_version: "encryptionkey@0"
I20260808 04:49:14.293434 5353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:49:14.295670 5381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:14.296850 5353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:14.296950 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/wal
uuid: "e6a413b32eeb49918801e9d08367d4e3"
format_stamp: "Formatted at 2026-08-08 04:49:14 on dist-test-slave-2kcd"
server_key: "a13017d752dab105fab79fdda2631c5d"
server_key_iv: "6094e8caece5722db0ea54d81e482e88"
server_key_version: "encryptionkey@0"
I20260808 04:49:14.297042 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:14.314740 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:14.315423 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:14.315580 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:14.316136 5353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:49:14.317130 5353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:49:14.317178 5353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:14.317221 5353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:49:14.317243 5353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:14.325904 5353 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:45585
I20260808 04:49:14.325935 5549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.3:45585 every 8 connection(s)
I20260808 04:49:14.326846 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/data/info.pb
I20260808 04:49:14.329130 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 5353
I20260808 04:49:14.329268 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/wal/instance
I20260808 04:49:14.329556 4964 external_mini_cluster.cc:1428] Setting key 8b1a3dfd78f09b2fd09db5f788493677
I20260808 04:49:14.336808 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.328738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.3:60321 (local address 127.4.217.62:41739)
0808 04:49:14.328962 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:14.328967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:14.330170 (+ 1203us) server_negotiation.cc:407] Connection header received
0808 04:49:14.331976 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:14.331979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:14.332038 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:14.332121 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:14.333054 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.333612 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.334787 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.335021 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.335136 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.336250 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:14.336254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:14.336257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:14.336270 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:14.336295 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:14.336303 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:14.336378 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:14.336562 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 04:49:14.336629 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260808 04:49:14.337706 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.329216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41739 (local address 127.4.217.3:60321)
0808 04:49:14.330006 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:14.330060 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:14.331743 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:14.332300 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:14.332307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:14.332391 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:14.332906 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.332919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.333750 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.333753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.334652 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.334662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.335181 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.335185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.335353 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.335982 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:49:14.335983 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:14.336025 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:49:14.336090 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:14.336393 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:14.336421 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:14.336426 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:49:14.336557 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 04:49:14.336879 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":145,"threads_started":1}
I20260808 04:49:14.338860 5550 heartbeater.cc:347] Connected to a master server at 127.4.217.62:41739
I20260808 04:49:14.339110 5550 heartbeater.cc:464] Registering TS with master...
I20260808 04:49:14.339658 5550 heartbeater.cc:511] Master 127.4.217.62:41739 requested a full tablet report, sending...
I20260808 04:49:14.340749 4835 ts_manager.cc:194] Registered new tserver with Master: e6a413b32eeb49918801e9d08367d4e3 (127.4.217.3:45585)
I20260808 04:49:14.341347 4835 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.217.3:60321
I20260808 04:49:14.343791 4964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:49:14.349956 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.346040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.4.217.62:41739)
0808 04:49:14.346200 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:14.346204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:14.346432 (+ 228us) server_negotiation.cc:407] Connection header received
0808 04:49:14.346597 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:14.346600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:14.346651 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:14.346738 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:14.347039 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.347548 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.348364 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.348600 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.348685 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.349585 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:14.349588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:14.349590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:14.349602 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:14.349621 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:14.349629 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:14.349699 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:14.349798 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 04:49:14.349847 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:49:14.353039 4835 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:49:14.356212 4835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:49:14.371568 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.365340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:45585 (local address 127.0.0.1:46394)
0808 04:49:14.366016 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:14.366028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:14.366101 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:14.367050 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:14.367053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:14.367067 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:14.367152 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.367162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.368684 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.368688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.369882 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.369893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.370954 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.370958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.371066 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.371104 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:49:14.371238 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 04:49:14.371392 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":22,"thread_start_us":110,"threads_started":1}
I20260808 04:49:14.371627 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.365768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.4.217.3:45585)
0808 04:49:14.366602 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:14.366616 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:14.366641 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:49:14.366719 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:14.366722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:14.366899 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:14.367305 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:14.367453 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.368540 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.369995 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.370793 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.370911 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.371104 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:14.371290 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 04:49:14.371402 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":185,"threads_started":1}
I20260808 04:49:14.371876 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.365105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:38775 (local address 127.0.0.1:56798)
0808 04:49:14.365850 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:14.365902 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:14.366053 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:14.367093 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:14.367096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:14.367114 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:14.367226 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.367233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.368216 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.368220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.370195 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.370200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.371214 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.371219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.371453 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.371477 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:49:14.371548 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:49:14.371646 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":81,"spinlock_wait_cycles":768,"thread_start_us":103,"threads_started":1}
I20260808 04:49:14.372084 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.366232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.4.217.1:38775)
0808 04:49:14.366660 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:14.366670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:14.366690 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:49:14.366764 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:14.366768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:14.366952 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:14.367087 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:14.367352 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.368089 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.370311 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.370819 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.370959 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.371143 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:14.371795 (+ 652us) server_negotiation.cc:299] Negotiation successful
0808 04:49:14.371901 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":117,"threads_started":1}
I20260808 04:49:14.372403 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.364923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.2:44583 (local address 127.0.0.1:46542)
0808 04:49:14.365878 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:14.365920 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:14.366054 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:14.366906 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:14.366913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:14.366948 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:14.367115 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.367125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.368611 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.368615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.369820 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.369841 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.371194 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.371197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.371372 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.371458 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 04:49:14.371536 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:49:14.372270 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":696,"thread_start_us":93,"threads_started":1}
I20260808 04:49:14.373032 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.365130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.4.217.2:44583)
0808 04:49:14.366390 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:14.366404 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:14.366430 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:49:14.366512 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:14.366518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:14.366733 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:14.366897 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:14.367266 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.368464 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.369992 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.370793 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.370911 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.371104 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:14.372725 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0808 04:49:14.372883 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":134,"threads_started":1}
I20260808 04:49:14.375243 5240 tablet_service.cc:1467] Processing CreateTablet for tablet 07ab349e1038457ca54fe8147c1830f0 (DEFAULT_TABLE table=test-table [id=11d2aec176e043659f1cc4e92cbc13ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:14.375218 5026 tablet_service.cc:1467] Processing CreateTablet for tablet 07ab349e1038457ca54fe8147c1830f0 (DEFAULT_TABLE table=test-table [id=11d2aec176e043659f1cc4e92cbc13ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:14.375218 5449 tablet_service.cc:1467] Processing CreateTablet for tablet 07ab349e1038457ca54fe8147c1830f0 (DEFAULT_TABLE table=test-table [id=11d2aec176e043659f1cc4e92cbc13ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:49:14.376250 5240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ab349e1038457ca54fe8147c1830f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:14.376271 5026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ab349e1038457ca54fe8147c1830f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:14.376271 5449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ab349e1038457ca54fe8147c1830f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:14.383095 5577 tablet_bootstrap.cc:492] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: Bootstrap starting.
I20260808 04:49:14.384465 5578 tablet_bootstrap.cc:492] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238: Bootstrap starting.
I20260808 04:49:14.384465 5579 tablet_bootstrap.cc:492] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: Bootstrap starting.
I20260808 04:49:14.385607 5577 tablet_bootstrap.cc:654] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:14.386415 5577 log.cc:824] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:14.386669 5579 tablet_bootstrap.cc:654] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:14.386669 5578 tablet_bootstrap.cc:654] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:14.387519 5578 log.cc:824] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:14.387519 5579 log.cc:824] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:14.388621 5577 tablet_bootstrap.cc:492] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: No bootstrap required, opened a new log
I20260808 04:49:14.388844 5577 ts_tablet_manager.cc:1397] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:49:14.389595 5578 tablet_bootstrap.cc:492] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238: No bootstrap required, opened a new log
I20260808 04:49:14.389592 5579 tablet_bootstrap.cc:492] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: No bootstrap required, opened a new log
I20260808 04:49:14.389807 5579 ts_tablet_manager.cc:1397] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:49:14.389816 5578 ts_tablet_manager.cc:1397] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:49:14.393078 5578 raft_consensus.cc:348] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } }
I20260808 04:49:14.393328 5578 raft_consensus.cc:374] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:14.393385 5578 raft_consensus.cc:729] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab3a2c00b884e748f99c48f05aa8238, State: Initialized, Role: FOLLOWER
I20260808 04:49:14.393270 5577 raft_consensus.cc:348] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } }
I20260808 04:49:14.393537 5577 raft_consensus.cc:374] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:14.393610 5577 raft_consensus.cc:729] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600e69544b254e23977f7d38d03c4013, State: Initialized, Role: FOLLOWER
I20260808 04:49:14.393898 5578 consensus_queue.cc:260] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } }
I20260808 04:49:14.394022 5577 consensus_queue.cc:260] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } }
I20260808 04:49:14.393947 5579 raft_consensus.cc:348] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } }
I20260808 04:49:14.394574 5130 heartbeater.cc:503] Master 127.4.217.62:41739 was elected leader, sending a full tablet report...
I20260808 04:49:14.394958 5578 ts_tablet_manager.cc:1428] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:49:14.394976 5577 ts_tablet_manager.cc:1428] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:49:14.394177 5579 raft_consensus.cc:374] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:14.395146 5579 raft_consensus.cc:729] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a413b32eeb49918801e9d08367d4e3, State: Initialized, Role: FOLLOWER
I20260808 04:49:14.395484 5579 consensus_queue.cc:260] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } }
I20260808 04:49:14.395757 5348 heartbeater.cc:503] Master 127.4.217.62:41739 was elected leader, sending a full tablet report...
I20260808 04:49:14.396121 5550 heartbeater.cc:503] Master 127.4.217.62:41739 was elected leader, sending a full tablet report...
I20260808 04:49:14.396404 5579 ts_tablet_manager.cc:1428] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260808 04:49:14.432565 5131 tablet.cc:2369] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:49:14.528888 5585 raft_consensus.cc:482] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:49:14.529168 5585 raft_consensus.cc:504] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } }
I20260808 04:49:14.530380 5585 leader_election.cc:288] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ab3a2c00b884e748f99c48f05aa8238 (127.4.217.1:38775), e6a413b32eeb49918801e9d08367d4e3 (127.4.217.3:45585)
I20260808 04:49:14.534330 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.530631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:38775 (local address 127.4.217.2:57245)
0808 04:49:14.530738 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:14.530754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:14.530850 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:14.531172 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:14.531175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:14.531197 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:14.531322 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.531328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.532272 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.532274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.533349 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.533360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.534024 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.534028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.534100 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.534120 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 04:49:14.534179 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 04:49:14.534228 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 04:49:14.534416 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.530746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:57245 (local address 127.4.217.1:38775)
0808 04:49:14.530864 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:14.530869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:14.530931 (+ 62us) server_negotiation.cc:407] Connection header received
0808 04:49:14.530993 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:14.530997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:14.531051 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:14.531125 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:14.531430 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.532156 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.533473 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.533910 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.534018 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.534082 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:14.534218 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 04:49:14.534290 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 04:49:14.535166 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ab349e1038457ca54fe8147c1830f0" candidate_uuid: "600e69544b254e23977f7d38d03c4013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab3a2c00b884e748f99c48f05aa8238" is_pre_election: true
I20260808 04:49:14.535344 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.530631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.3:45585 (local address 127.4.217.2:45629)
0808 04:49:14.531547 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:14.531557 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:14.531627 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:14.531845 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:14.531847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:14.531859 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:49:14.531949 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.531952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.532737 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.532741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.533756 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:14.533764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.534927 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.534931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:14.535027 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.535049 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:49:14.535131 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:49:14.535204 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":270,"thread_start_us":179,"threads_started":1}
I20260808 04:49:14.535342 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.530842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.217.2:45629 (local address 127.4.217.3:45585)
0808 04:49:14.530987 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:14.530992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:14.531638 (+ 646us) server_negotiation.cc:407] Connection header received
0808 04:49:14.531700 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:14.531704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:14.531750 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:14.531828 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:49:14.532050 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.532628 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.533877 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.534492 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.534598 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.534640 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:14.535163 (+ 523us) server_negotiation.cc:299] Negotiation successful
0808 04:49:14.535222 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:49:14.535482 5063 raft_consensus.cc:2454] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600e69544b254e23977f7d38d03c4013 in term 0.
I20260808 04:49:14.536018 5164 leader_election.cc:302] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ab3a2c00b884e748f99c48f05aa8238, 600e69544b254e23977f7d38d03c4013; no voters:
I20260808 04:49:14.536041 5482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ab349e1038457ca54fe8147c1830f0" candidate_uuid: "600e69544b254e23977f7d38d03c4013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a413b32eeb49918801e9d08367d4e3" is_pre_election: true
I20260808 04:49:14.536275 5585 raft_consensus.cc:2781] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:49:14.536327 5482 raft_consensus.cc:2454] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600e69544b254e23977f7d38d03c4013 in term 0.
I20260808 04:49:14.536379 5585 raft_consensus.cc:482] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:49:14.536427 5585 raft_consensus.cc:3037] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:14.537850 5585 raft_consensus.cc:504] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } }
I20260808 04:49:14.538165 5585 leader_election.cc:288] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [CANDIDATE]: Term 1 election: Requested vote from peers 5ab3a2c00b884e748f99c48f05aa8238 (127.4.217.1:38775), e6a413b32eeb49918801e9d08367d4e3 (127.4.217.3:45585)
I20260808 04:49:14.538483 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ab349e1038457ca54fe8147c1830f0" candidate_uuid: "600e69544b254e23977f7d38d03c4013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab3a2c00b884e748f99c48f05aa8238"
I20260808 04:49:14.538509 5482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ab349e1038457ca54fe8147c1830f0" candidate_uuid: "600e69544b254e23977f7d38d03c4013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a413b32eeb49918801e9d08367d4e3"
I20260808 04:49:14.538636 5482 raft_consensus.cc:3037] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:14.538650 5063 raft_consensus.cc:3037] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:14.539659 5482 raft_consensus.cc:2454] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600e69544b254e23977f7d38d03c4013 in term 1.
I20260808 04:49:14.539697 5063 raft_consensus.cc:2454] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600e69544b254e23977f7d38d03c4013 in term 1.
I20260808 04:49:14.539973 5165 leader_election.cc:302] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 600e69544b254e23977f7d38d03c4013, e6a413b32eeb49918801e9d08367d4e3; no voters:
I20260808 04:49:14.540174 5585 raft_consensus.cc:2781] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:14.540416 5585 raft_consensus.cc:686] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 1 LEADER]: Becoming Leader. State: Replica: 600e69544b254e23977f7d38d03c4013, State: Running, Role: LEADER
I20260808 04:49:14.540756 5585 consensus_queue.cc:237] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } }
I20260808 04:49:14.543999 4835 catalog_manager.cc:5495] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600e69544b254e23977f7d38d03c4013 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "600e69544b254e23977f7d38d03c4013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600e69544b254e23977f7d38d03c4013" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 } health_report { overall_health: UNKNOWN } } }
I20260808 04:49:14.560038 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.556461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.4.217.2:44583)
0808 04:49:14.556625 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:14.556628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:14.556639 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:49:14.556709 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:14.556711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:14.556765 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:14.556944 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:14.557195 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.557710 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.558571 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:14.558790 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:14.558897 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:14.559220 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:14.559630 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:14.559760 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:14.559830 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 04:49:14.559884 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:49:14.566488 5482 raft_consensus.cc:1261] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [term 1 FOLLOWER]: Refusing update from remote peer 600e69544b254e23977f7d38d03c4013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:14.566498 5063 raft_consensus.cc:1261] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238 [term 1 FOLLOWER]: Refusing update from remote peer 600e69544b254e23977f7d38d03c4013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:14.567250 5585 consensus_queue.cc:1035] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6a413b32eeb49918801e9d08367d4e3" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:14.567476 5591 consensus_queue.cc:1035] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ab3a2c00b884e748f99c48f05aa8238" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 38775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:14.576771 5596 mvcc.cc:205] Tried to move back new op lower bound from 7316130015496572928 to 7316130015401783296. Current Snapshot: MvccSnapshot[applied={T|T < 7316130015496572928}]
W20260808 04:49:14.577795 5551 tablet.cc:2369] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:49:14.578641 5599 mvcc.cc:205] Tried to move back new op lower bound from 7316130015496572928 to 7316130015401783296. Current Snapshot: MvccSnapshot[applied={T|T < 7316130015496572928}]
I20260808 04:49:14.585979 4835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:14.586160 4835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:49:14.588739 4835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ddc9e83ef3845148bdb4c28a7a2152d: Sending DeleteTablet for 3 replicas of tablet 07ab349e1038457ca54fe8147c1830f0
I20260808 04:49:14.589615 5240 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ab349e1038457ca54fe8147c1830f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:14 UTC) from {username='slave'} at 127.0.0.1:46542
I20260808 04:49:14.589689 5026 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ab349e1038457ca54fe8147c1830f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:14 UTC) from {username='slave'} at 127.0.0.1:56798
I20260808 04:49:14.590130 5612 tablet_replica.cc:330] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: stopping tablet replica
I20260808 04:49:14.590435 5612 raft_consensus.cc:2229] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:49:14.590445 5449 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ab349e1038457ca54fe8147c1830f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:49:14 UTC) from {username='slave'} at 127.0.0.1:46394
I20260808 04:49:14.590601 5613 tablet_replica.cc:330] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238: stopping tablet replica
I20260808 04:49:14.590689 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 4921
I20260808 04:49:14.590812 5612 raft_consensus.cc:2258] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:14.590973 5613 raft_consensus.cc:2229] T 07ab349e1038457ca54fe8147c1830f0 P 5ab3a2c00b884e748f99c48f05aa8238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:14.592368 5612 ts_tablet_manager.cc:1905] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:14.596092 5612 ts_tablet_manager.cc:1918] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:14.596235 5612 log.cc:1196] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-1/wal/wals/07ab349e1038457ca54fe8147c1830f0
I20260808 04:49:14.596603 5612 ts_tablet_manager.cc:1939] T 07ab349e1038457ca54fe8147c1830f0 P 600e69544b254e23977f7d38d03c4013: Deleting consensus metadata
I20260808 04:49:14.599376 4810 catalog_manager.cc:4841] TS 600e69544b254e23977f7d38d03c4013 (127.4.217.2:44583): tablet 07ab349e1038457ca54fe8147c1830f0 (table test-table [id=11d2aec176e043659f1cc4e92cbc13ca]) successfully deleted
I20260808 04:49:14.600534 5615 tablet_replica.cc:330] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: stopping tablet replica
I20260808 04:49:14.600970 5615 raft_consensus.cc:2229] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:49:14.601351 5615 raft_consensus.cc:2258] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:49:14.607434 5615 ts_tablet_manager.cc:1905] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:49:14.610425 5615 ts_tablet_manager.cc:1918] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:49:14.610544 5615 log.cc:1196] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: Deleting WAL directory at /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164296366965-4964-0/minicluster-data/ts-2/wal/wals/07ab349e1038457ca54fe8147c1830f0
I20260808 04:49:14.610976 5615 ts_tablet_manager.cc:1939] T 07ab349e1038457ca54fe8147c1830f0 P e6a413b32eeb49918801e9d08367d4e3: Deleting consensus metadata
W20260808 04:49:14.612532 4813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.217.1:38775 (error 108)
I20260808 04:49:14.612546 4815 catalog_manager.cc:4841] TS e6a413b32eeb49918801e9d08367d4e3 (127.4.217.3:45585): tablet 07ab349e1038457ca54fe8147c1830f0 (table test-table [id=11d2aec176e043659f1cc4e92cbc13ca]) successfully deleted
I20260808 04:49:14.613577 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 5135
I20260808 04:49:14.613812 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:14.613456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.1:38775 (local address 127.0.0.1:56814)
0808 04:49:14.613615 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:14.613703 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.1:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 04:49:14.620151 4813 catalog_manager.cc:4584] TS 5ab3a2c00b884e748f99c48f05aa8238 (127.4.217.1:38775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07ab349e1038457ca54fe8147c1830f0: Network error: Client connection negotiation failed: client connection to 127.4.217.1:38775: connect: Connection refused (error 111)
I20260808 04:49:14.641031 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 5353
I20260808 04:49:14.657909 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 4790
2026-08-08T04:49:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1193 ms)
[----------] 6 tests from EncryptionPolicyTest (11106 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:14 dist-test-slave-2kcd krb5kdc[5623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:14 dist-test-slave-2kcd krb5kdc[5623](info): set up 2 sockets
Aug 08 04:49:14 dist-test-slave-2kcd krb5kdc[5623](info): commencing operation
krb5kdc: starting...
W20260808 04:49:15.928619 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:16 dist-test-slave-2kcd krb5kdc[5623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:16.055218 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:16.167421 5642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:16.167651 5642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:16.167689 5642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:16.171633 5642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:16.171686 5642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:16.171705 5642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:16.171759 5642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:16.171785 5642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:16.176723 5642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:16.177716 5642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:16.178700 5642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:16.187098 5653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:16.187057 5661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:16.187129 5664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:16.187631 5642 server_base.cc:1034] running on GCE node
I20260808 04:49:16.188140 5642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:16.189246 5642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:16.190441 5642 hybrid_clock.cc:648] HybridClock initialized: now 1786164556190432 us; error 29 us; skew 500 ppm
Aug 08 04:49:16 dist-test-slave-2kcd krb5kdc[5623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:16.193104 5642 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:16.193977 5642 webserver.cc:460] Webserver started at http://127.0.0.1:38537/ using document root <none> and password file <none>
I20260808 04:49:16.194526 5642 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:16.194619 5642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:16.194852 5642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:16.196877 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "f668e2edebc14f049717c450d8c7bc85"
format_stamp: "Formatted at 2026-08-08 04:49:16 on dist-test-slave-2kcd"
server_key: "c4ff745ee255c65e48cc77d892afbb1b"
server_key_iv: "ef2fb38a7d58e6dd1ce8c2626cd6dffc"
server_key_version: "encryptionkey@0"
I20260808 04:49:16.197441 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "f668e2edebc14f049717c450d8c7bc85"
format_stamp: "Formatted at 2026-08-08 04:49:16 on dist-test-slave-2kcd"
server_key: "c4ff745ee255c65e48cc77d892afbb1b"
server_key_iv: "ef2fb38a7d58e6dd1ce8c2626cd6dffc"
server_key_version: "encryptionkey@0"
I20260808 04:49:16.201354 5642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260808 04:49:16.203637 5674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:16.204731 5642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:16.204898 5642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "f668e2edebc14f049717c450d8c7bc85"
format_stamp: "Formatted at 2026-08-08 04:49:16 on dist-test-slave-2kcd"
server_key: "c4ff745ee255c65e48cc77d892afbb1b"
server_key_iv: "ef2fb38a7d58e6dd1ce8c2626cd6dffc"
server_key_version: "encryptionkey@0"
I20260808 04:49:16.205032 5642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:16.234323 5642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:16.236956 5642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:16.237161 5642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:16.243983 5642 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39449
I20260808 04:49:16.244005 5753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39449 every 8 connection(s)
I20260808 04:49:16.244901 5642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:16.247495 5754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:16.252457 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 5642
I20260808 04:49:16.252640 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:16.253021 4964 external_mini_cluster.cc:1428] Setting key eed55e74c87fec7462e65df2b8859131
I20260808 04:49:16.254966 5754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85: Bootstrap starting.
I20260808 04:49:16.257668 5754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:16.258592 5754 log.cc:824] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:16.260599 5754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85: No bootstrap required, opened a new log
Aug 08 04:49:16 dist-test-slave-2kcd krb5kdc[5623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:49:16.264909 5754 raft_consensus.cc:348] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f668e2edebc14f049717c450d8c7bc85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } }
I20260808 04:49:16.265143 5754 raft_consensus.cc:374] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:16.265208 5754 raft_consensus.cc:729] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f668e2edebc14f049717c450d8c7bc85, State: Initialized, Role: FOLLOWER
I20260808 04:49:16.265677 5754 consensus_queue.cc:260] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f668e2edebc14f049717c450d8c7bc85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } }
I20260808 04:49:16.265796 5754 raft_consensus.cc:388] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:16.265879 5754 raft_consensus.cc:482] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:16.266005 5754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:16.266839 5754 raft_consensus.cc:504] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f668e2edebc14f049717c450d8c7bc85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } }
I20260808 04:49:16.267195 5754 leader_election.cc:302] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f668e2edebc14f049717c450d8c7bc85; no voters:
I20260808 04:49:16.267527 5754 leader_election.cc:288] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:16.267592 5765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:16.267818 5765 raft_consensus.cc:686] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [term 1 LEADER]: Becoming Leader. State: Replica: f668e2edebc14f049717c450d8c7bc85, State: Running, Role: LEADER
I20260808 04:49:16.268119 5765 consensus_queue.cc:237] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f668e2edebc14f049717c450d8c7bc85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } }
I20260808 04:49:16.268869 5754 sys_catalog.cc:564] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:16.269351 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:16.254644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41657 (local address 127.0.0.1:39449)
0808 04:49:16.255200 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:16.255220 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:16.255261 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:49:16.256062 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:16.256090 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:16.256511 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:16.256870 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:16.257960 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:16.259340 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:16.260194 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:16.260885 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:16.261060 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:16.263815 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:16.263821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:16.263833 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:16.263861 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:16.265799 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:16.266364 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:16.266370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:16.266376 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:16.266496 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:16.266840 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:16.266843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:16.266845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:16.267285 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:16.267523 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:16.268142 (+ 619us) server_negotiation.cc:299] Negotiation successful
0808 04:49:16.268377 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":116,"threads_started":1}
I20260808 04:49:16.269335 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f668e2edebc14f049717c450d8c7bc85. Latest consensus state: current_term: 1 leader_uuid: "f668e2edebc14f049717c450d8c7bc85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f668e2edebc14f049717c450d8c7bc85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } } }
I20260808 04:49:16.269500 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:16.269905 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f668e2edebc14f049717c450d8c7bc85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f668e2edebc14f049717c450d8c7bc85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } } }
I20260808 04:49:16.270007 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:16.270669 5773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:16.272923 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:16.273764 5773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:16.280189 5773 catalog_manager.cc:1347] Generated new cluster ID: a02c2e1303f2473bbce5ac67fa903ba7
I20260808 04:49:16.280251 5773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:16.284147 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:16.275804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39677 (local address 127.0.0.1:39449)
0808 04:49:16.275910 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:16.275915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:16.276160 (+ 245us) server_negotiation.cc:407] Connection header received
0808 04:49:16.276296 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:16.276299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:16.276347 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:16.276436 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:16.277131 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:16.277634 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:16.278582 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:16.278832 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:16.278944 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:16.280458 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:16.280469 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:16.280474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:16.280504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:16.282281 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:16.282797 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:16.282802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:16.282806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:16.282899 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:16.283230 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:16.283233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:16.283235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:16.283447 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:16.283603 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:16.283885 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 04:49:16.283982 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:49:16.304243 5773 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:16.305606 5773 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:16.313942 5773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f668e2edebc14f049717c450d8c7bc85: Generated new TSK 0
I20260808 04:49:16.314745 5773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:16.322378 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 5642
W20260808 04:49:16.339371 5618 connection.cc:535] client connection to 127.0.0.1:39449 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39449): Connection reset by peer (error 104)
2026-08-08T04:49:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1666 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:16 dist-test-slave-2kcd krb5kdc[5805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:16 dist-test-slave-2kcd krb5kdc[5805](info): set up 2 sockets
Aug 08 04:49:16 dist-test-slave-2kcd krb5kdc[5805](info): commencing operation
krb5kdc: starting...
W20260808 04:49:17.596271 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:17 dist-test-slave-2kcd krb5kdc[5805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:17.719745 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:17.832511 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:17.832796 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:17.832906 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:17.836964 5825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:17.837030 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:17.837050 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:17.837066 5825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:17.837081 5825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:17.842365 5825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:17.843459 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:17.844514 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:49:17.853266 5825 server_base.cc:1034] running on GCE node
W20260808 04:49:17.853330 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:17.853288 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:17.853241 5840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:17.854041 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:17.855111 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:17.856293 5825 hybrid_clock.cc:648] HybridClock initialized: now 1786164557856263 us; error 39 us; skew 500 ppm
Aug 08 04:49:17 dist-test-slave-2kcd krb5kdc[5805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:17.859114 5825 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:17.860000 5825 webserver.cc:460] Webserver started at http://127.0.0.1:36685/ using document root <none> and password file <none>
I20260808 04:49:17.860530 5825 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:17.860589 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:17.860752 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:17.862856 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "ce09d6797bff4e30b83485c7db239ff0"
format_stamp: "Formatted at 2026-08-08 04:49:17 on dist-test-slave-2kcd"
server_key: "d7c3a93fba5d60adb2fb51b076b2c09a"
server_key_iv: "a152ec8429d1a3d85ac5cd54c6d3ccd7"
server_key_version: "encryptionkey@0"
I20260808 04:49:17.863379 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "ce09d6797bff4e30b83485c7db239ff0"
format_stamp: "Formatted at 2026-08-08 04:49:17 on dist-test-slave-2kcd"
server_key: "d7c3a93fba5d60adb2fb51b076b2c09a"
server_key_iv: "a152ec8429d1a3d85ac5cd54c6d3ccd7"
server_key_version: "encryptionkey@0"
I20260808 04:49:17.867097 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260808 04:49:17.869556 5852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:17.870913 5825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:49:17.871040 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "ce09d6797bff4e30b83485c7db239ff0"
format_stamp: "Formatted at 2026-08-08 04:49:17 on dist-test-slave-2kcd"
server_key: "d7c3a93fba5d60adb2fb51b076b2c09a"
server_key_iv: "a152ec8429d1a3d85ac5cd54c6d3ccd7"
server_key_version: "encryptionkey@0"
I20260808 04:49:17.871135 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:17.895625 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:17.898638 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:17.898802 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:17.906287 5825 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38533
I20260808 04:49:17.906322 5937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38533 every 8 connection(s)
I20260808 04:49:17.907261 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:17.910054 5939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:17.915585 5939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0: Bootstrap starting.
I20260808 04:49:17.916760 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 5825
I20260808 04:49:17.916944 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:17.917284 4964 external_mini_cluster.cc:1428] Setting key fde9831590774a8798d17b9a5c98eab0
I20260808 04:49:17.918668 5939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:17.919413 5939 log.cc:824] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:17.921553 5939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0: No bootstrap required, opened a new log
I20260808 04:49:17.925977 5939 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce09d6797bff4e30b83485c7db239ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } }
I20260808 04:49:17.926237 5939 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:17.926319 5939 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce09d6797bff4e30b83485c7db239ff0, State: Initialized, Role: FOLLOWER
I20260808 04:49:17.926508 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:17.918450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59581 (local address 127.0.0.1:38533)
0808 04:49:17.918887 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:17.918904 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:17.918941 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:49:17.919633 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:17.919652 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:17.920078 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:17.920414 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:17.920745 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:17.921823 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:17.922979 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:17.923442 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:17.923562 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:17.924519 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:17.924525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:17.924537 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:17.925030 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:17.925089 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:17.925113 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:17.925220 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:17.925465 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 04:49:17.925618 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":141,"threads_started":1}
I20260808 04:49:17.926849 5939 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce09d6797bff4e30b83485c7db239ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } }
I20260808 04:49:17.927014 5939 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:17.927073 5939 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:17.927170 5939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:17.928156 5939 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce09d6797bff4e30b83485c7db239ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } }
I20260808 04:49:17.928532 5939 leader_election.cc:302] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce09d6797bff4e30b83485c7db239ff0; no voters:
I20260808 04:49:17.928954 5939 leader_election.cc:288] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:17.929028 5948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:17.929256 5948 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [term 1 LEADER]: Becoming Leader. State: Replica: ce09d6797bff4e30b83485c7db239ff0, State: Running, Role: LEADER
I20260808 04:49:17.929564 5948 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce09d6797bff4e30b83485c7db239ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } }
I20260808 04:49:17.929785 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:17.930291 5939 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:17.931203 5951 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce09d6797bff4e30b83485c7db239ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce09d6797bff4e30b83485c7db239ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } } }
I20260808 04:49:17.931339 5951 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:17.931542 5950 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce09d6797bff4e30b83485c7db239ff0. Latest consensus state: current_term: 1 leader_uuid: "ce09d6797bff4e30b83485c7db239ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce09d6797bff4e30b83485c7db239ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } } }
I20260808 04:49:17.931633 5950 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:17.933212 5969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:17.936002 5969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:17.937163 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:17.932203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54967 (local address 127.0.0.1:38533)
0808 04:49:17.932323 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:17.932329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:17.932821 (+ 492us) server_negotiation.cc:407] Connection header received
0808 04:49:17.932988 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:17.932991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:17.933041 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:17.933110 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:17.933557 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:17.934080 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:17.935252 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:17.935584 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:17.935697 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:17.936639 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:17.936643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:17.936645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:17.936659 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:17.936688 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:17.936700 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:17.936839 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:17.937001 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 04:49:17.937036 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:49:17.941776 5969 catalog_manager.cc:1347] Generated new cluster ID: 111ea6ade5374476b12827988847e6df
I20260808 04:49:17.941846 5969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:17.956202 5969 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:17.957455 5969 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:17.965783 5969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce09d6797bff4e30b83485c7db239ff0: Generated new TSK 0
I20260808 04:49:17.966531 5969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:17.971333 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 5825
W20260808 04:49:17.989885 5802 connection.cc:535] client connection to 127.0.0.1:38533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38533): Connection reset by peer (error 104)
2026-08-08T04:49:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1650 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:18 dist-test-slave-2kcd krb5kdc[5991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:18 dist-test-slave-2kcd krb5kdc[5991](info): set up 2 sockets
Aug 08 04:49:18 dist-test-slave-2kcd krb5kdc[5991](info): commencing operation
krb5kdc: starting...
W20260808 04:49:19.236250 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:19 dist-test-slave-2kcd krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:19.361815 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:19.473464 6009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:19.473711 6009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:19.473750 6009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:19.477528 6009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:19.477574 6009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:19.477593 6009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:19.477608 6009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:19.477623 6009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:19.482666 6009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:19.483556 6009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:19.484494 6009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:19.492998 6018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:19.493044 6009 server_base.cc:1034] running on GCE node
W20260808 04:49:19.493011 6014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:19.493006 6015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:19.493723 6009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:19.494760 6009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:19.495949 6009 hybrid_clock.cc:648] HybridClock initialized: now 1786164559495925 us; error 45 us; skew 500 ppm
Aug 08 04:49:19 dist-test-slave-2kcd krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:19.506632 6009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:19.507674 6009 webserver.cc:460] Webserver started at http://127.0.0.1:33463/ using document root <none> and password file <none>
I20260808 04:49:19.508276 6009 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:19.508391 6009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:19.508670 6009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:19.510932 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "5db48cd2be4c4d41a50e9ba0d4265246"
format_stamp: "Formatted at 2026-08-08 04:49:19 on dist-test-slave-2kcd"
server_key: "81170e2db164707328278309dc4860e2"
server_key_iv: "d41e67d6f6c3111751551a4e7020ca61"
server_key_version: "encryptionkey@0"
I20260808 04:49:19.511535 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "5db48cd2be4c4d41a50e9ba0d4265246"
format_stamp: "Formatted at 2026-08-08 04:49:19 on dist-test-slave-2kcd"
server_key: "81170e2db164707328278309dc4860e2"
server_key_iv: "d41e67d6f6c3111751551a4e7020ca61"
server_key_version: "encryptionkey@0"
I20260808 04:49:19.515609 6009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:49:19.518289 6031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:19.519548 6009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:49:19.519701 6009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "5db48cd2be4c4d41a50e9ba0d4265246"
format_stamp: "Formatted at 2026-08-08 04:49:19 on dist-test-slave-2kcd"
server_key: "81170e2db164707328278309dc4860e2"
server_key_iv: "d41e67d6f6c3111751551a4e7020ca61"
server_key_version: "encryptionkey@0"
I20260808 04:49:19.519820 6009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:19.524550 6009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:19.527874 6009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:19.528074 6009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:19.535320 6009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39179
I20260808 04:49:19.535365 6117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39179 every 8 connection(s)
I20260808 04:49:19.536284 6009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:19.538158 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6009
I20260808 04:49:19.538307 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:19.538637 4964 external_mini_cluster.cc:1428] Setting key ab3d24079b4e5a59020da923f6624ac8
I20260808 04:49:19.539569 6118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:19.543954 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:19.540075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50459 (local address 127.0.0.1:39179)
0808 04:49:19.540497 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:19.540513 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:19.540549 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:49:19.541280 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:19.541298 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:19.541586 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:19.541858 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:19.542725 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:19.542730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:19.542741 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:19.542751 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:19.542802 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:19.542821 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:19.542937 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:19.543063 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 04:49:19.543232 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":142,"threads_started":1}
I20260808 04:49:19.546416 6118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246: Bootstrap starting.
I20260808 04:49:19.547917 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:19.549042 6118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:19.549816 6118 log.cc:824] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:19.551784 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:19.550151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33855 (local address 127.0.0.1:39179)
0808 04:49:19.550275 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:19.550280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:19.550510 (+ 230us) server_negotiation.cc:407] Connection header received
0808 04:49:19.550645 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:19.550648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:19.550677 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:19.550764 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:19.551429 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:19.551433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:19.551435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:19.551447 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:19.551474 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:19.551488 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:19.551575 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:19.551635 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 04:49:19.551672 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:49:19.552174 6118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246: No bootstrap required, opened a new log
I20260808 04:49:19.555714 6118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db48cd2be4c4d41a50e9ba0d4265246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } }
I20260808 04:49:19.555959 6118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:19.556025 6118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db48cd2be4c4d41a50e9ba0d4265246, State: Initialized, Role: FOLLOWER
I20260808 04:49:19.556515 6118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db48cd2be4c4d41a50e9ba0d4265246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } }
I20260808 04:49:19.556706 6118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:19.556825 6118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:19.556945 6118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:19.557806 6118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db48cd2be4c4d41a50e9ba0d4265246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } }
I20260808 04:49:19.558144 6118 leader_election.cc:302] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5db48cd2be4c4d41a50e9ba0d4265246; no voters:
I20260808 04:49:19.558457 6118 leader_election.cc:288] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:19.558586 6137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:19.558816 6137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [term 1 LEADER]: Becoming Leader. State: Replica: 5db48cd2be4c4d41a50e9ba0d4265246, State: Running, Role: LEADER
I20260808 04:49:19.559125 6137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db48cd2be4c4d41a50e9ba0d4265246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } }
I20260808 04:49:19.559716 6118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:19.560760 6138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5db48cd2be4c4d41a50e9ba0d4265246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db48cd2be4c4d41a50e9ba0d4265246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } } }
I20260808 04:49:19.560930 6138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:19.560889 6139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5db48cd2be4c4d41a50e9ba0d4265246. Latest consensus state: current_term: 1 leader_uuid: "5db48cd2be4c4d41a50e9ba0d4265246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db48cd2be4c4d41a50e9ba0d4265246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } } }
I20260808 04:49:19.561026 6139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:19.562189 6149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:19.565196 6149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:19.570766 6149 catalog_manager.cc:1347] Generated new cluster ID: 10a7d2301bcf449e9917c62249a5ba7f
I20260808 04:49:19.570842 6149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:19.600327 6149 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:19.600708 6149 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:19.609584 6149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5db48cd2be4c4d41a50e9ba0d4265246: Generated new TSK 0
I20260808 04:49:19.610244 6149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:19.616515 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6009
2026-08-08T04:49:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1643 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:19 dist-test-slave-2kcd krb5kdc[6163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:19 dist-test-slave-2kcd krb5kdc[6163](info): set up 2 sockets
Aug 08 04:49:19 dist-test-slave-2kcd krb5kdc[6163](info): commencing operation
krb5kdc: starting...
W20260808 04:49:20.888469 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:20 dist-test-slave-2kcd krb5kdc[6163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:21.015702 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:21.128057 6189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:21.128309 6189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:21.128350 6189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:21.132234 6189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:21.132285 6189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:21.132304 6189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:21.132320 6189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:21.132334 6189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:21.137313 6189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:21.138264 6189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:21.139199 6189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:49:21.147719 6189 server_base.cc:1034] running on GCE node
W20260808 04:49:21.147665 6198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:21.147650 6202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:21.147672 6197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:21.148476 6189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:21.149649 6189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:21.150811 6189 hybrid_clock.cc:648] HybridClock initialized: now 1786164561150783 us; error 39 us; skew 500 ppm
Aug 08 04:49:21 dist-test-slave-2kcd krb5kdc[6163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:21.153600 6189 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:21.154507 6189 webserver.cc:460] Webserver started at http://127.0.0.1:41325/ using document root <none> and password file <none>
I20260808 04:49:21.155066 6189 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:21.155158 6189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:21.155395 6189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:21.157456 6189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "dc407d2b621b4166addbb403ec7f423a"
format_stamp: "Formatted at 2026-08-08 04:49:21 on dist-test-slave-2kcd"
server_key: "ccae1d7c5b3d4791ab77596969b639fd"
server_key_iv: "d37d5660cf9c9c90af54fc0f35afaa8e"
server_key_version: "encryptionkey@0"
I20260808 04:49:21.158032 6189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "dc407d2b621b4166addbb403ec7f423a"
format_stamp: "Formatted at 2026-08-08 04:49:21 on dist-test-slave-2kcd"
server_key: "ccae1d7c5b3d4791ab77596969b639fd"
server_key_iv: "d37d5660cf9c9c90af54fc0f35afaa8e"
server_key_version: "encryptionkey@0"
I20260808 04:49:21.161649 6189 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:49:21.164083 6210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:21.165328 6189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:21.165480 6189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "dc407d2b621b4166addbb403ec7f423a"
format_stamp: "Formatted at 2026-08-08 04:49:21 on dist-test-slave-2kcd"
server_key: "ccae1d7c5b3d4791ab77596969b639fd"
server_key_iv: "d37d5660cf9c9c90af54fc0f35afaa8e"
server_key_version: "encryptionkey@0"
I20260808 04:49:21.165592 6189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:21.180745 6189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:21.183701 6189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:21.183908 6189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:21.191104 6189 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37115
I20260808 04:49:21.191129 6290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37115 every 8 connection(s)
I20260808 04:49:21.192039 6189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:21.192646 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6189
I20260808 04:49:21.192842 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:21.193184 4964 external_mini_cluster.cc:1428] Setting key e684375671176dbb815d7343439c13d7
I20260808 04:49:21.195323 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:21.201944 6293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a: Bootstrap starting.
Aug 08 04:49:21 dist-test-slave-2kcd krb5kdc[6163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:49:21.204681 6293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:21.205627 6293 log.cc:824] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:21.207744 6293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a: No bootstrap required, opened a new log
I20260808 04:49:21.207965 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:21.194567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45767 (local address 127.0.0.1:37115)
0808 04:49:21.194980 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:21.195002 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:21.195047 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:49:21.195904 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:21.195927 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:21.196327 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:21.196677 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:21.197417 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:21.198930 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:21.199731 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:21.200418 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:21.200593 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:21.203027 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:21.203037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:21.203053 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:21.203084 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:21.205060 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:21.205477 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:21.205486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:21.205494 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:21.205637 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:21.205945 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:21.205950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:21.205951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:21.206400 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:21.206595 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:21.206804 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 04:49:21.207079 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260808 04:49:21.210951 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:21.211861 6293 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc407d2b621b4166addbb403ec7f423a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37115 } }
I20260808 04:49:21.212148 6293 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:21.212245 6293 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc407d2b621b4166addbb403ec7f423a, State: Initialized, Role: FOLLOWER
I20260808 04:49:21.212711 6293 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc407d2b621b4166addbb403ec7f423a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37115 } }
I20260808 04:49:21.212893 6293 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:21.212997 6293 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:21.213150 6293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:21.214143 6293 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc407d2b621b4166addbb403ec7f423a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37115 } }
I20260808 04:49:21.214500 6293 leader_election.cc:302] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc407d2b621b4166addbb403ec7f423a; no voters:
I20260808 04:49:21.214790 6293 leader_election.cc:288] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:21.214910 6310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:21.215178 6310 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [term 1 LEADER]: Becoming Leader. State: Replica: dc407d2b621b4166addbb403ec7f423a, State: Running, Role: LEADER
I20260808 04:49:21.215539 6310 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc407d2b621b4166addbb403ec7f423a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37115 } }
I20260808 04:49:21.216054 6293 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:21.216835 6310 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc407d2b621b4166addbb403ec7f423a. Latest consensus state: current_term: 1 leader_uuid: "dc407d2b621b4166addbb403ec7f423a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc407d2b621b4166addbb403ec7f423a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37115 } } }
I20260808 04:49:21.217025 6310 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:21.217525 6323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:21.217525 6312 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc407d2b621b4166addbb403ec7f423a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc407d2b621b4166addbb403ec7f423a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37115 } } }
I20260808 04:49:21.217690 6312 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:21.221026 6323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:21.221767 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:21.212953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57459 (local address 127.0.0.1:37115)
0808 04:49:21.213099 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:21.213103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:21.213272 (+ 169us) server_negotiation.cc:407] Connection header received
0808 04:49:21.213374 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:21.213380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:21.213431 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:21.213529 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:21.214109 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:21.214858 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:21.215986 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:21.216329 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:21.216436 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:21.218030 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:21.218034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:21.218036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:21.218060 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:21.219722 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:21.220174 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:21.220178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:21.220179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:21.220265 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:21.220547 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:21.220550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:21.220552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:21.220757 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:21.220928 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:21.221503 (+ 575us) server_negotiation.cc:299] Negotiation successful
0808 04:49:21.221616 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:49:21.227213 6323 catalog_manager.cc:1347] Generated new cluster ID: 07ce985c1e864e3c99693a65398d65e1
I20260808 04:49:21.227281 6323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:21.236801 6323 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:21.237604 6323 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:21.244042 6323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc407d2b621b4166addbb403ec7f423a: Generated new TSK 0
I20260808 04:49:21.244691 6323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:21.248469 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6189
2026-08-08T04:49:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1631 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:21 dist-test-slave-2kcd krb5kdc[6345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:21 dist-test-slave-2kcd krb5kdc[6345](info): set up 2 sockets
Aug 08 04:49:21 dist-test-slave-2kcd krb5kdc[6345](info): commencing operation
krb5kdc: starting...
W20260808 04:49:22.530911 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:22 dist-test-slave-2kcd krb5kdc[6345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:22.657968 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:22.773790 6359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:22.774040 6359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:22.774116 6359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:22.778113 6359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:22.778167 6359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:22.778184 6359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:22.778200 6359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:22.778215 6359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:22.783382 6359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:22.784482 6359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:22.785513 6359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:22.794098 6366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:22.794174 6367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:22.794073 6372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:22.794464 6359 server_base.cc:1034] running on GCE node
I20260808 04:49:22.795007 6359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:22.796097 6359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:22.797307 6359 hybrid_clock.cc:648] HybridClock initialized: now 1786164562797287 us; error 26 us; skew 500 ppm
Aug 08 04:49:22 dist-test-slave-2kcd krb5kdc[6345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:22.800019 6359 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:22.800959 6359 webserver.cc:460] Webserver started at http://127.0.0.1:41159/ using document root <none> and password file <none>
I20260808 04:49:22.801530 6359 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:22.801622 6359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:22.801875 6359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:22.803987 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "29c911b96f084c6b8778a4b052668a5b"
format_stamp: "Formatted at 2026-08-08 04:49:22 on dist-test-slave-2kcd"
server_key: "eade836d2895b827d3390d8093456e09"
server_key_iv: "c34b357515996b2356a668dde35864b8"
server_key_version: "encryptionkey@0"
I20260808 04:49:22.804557 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "29c911b96f084c6b8778a4b052668a5b"
format_stamp: "Formatted at 2026-08-08 04:49:22 on dist-test-slave-2kcd"
server_key: "eade836d2895b827d3390d8093456e09"
server_key_iv: "c34b357515996b2356a668dde35864b8"
server_key_version: "encryptionkey@0"
I20260808 04:49:22.808248 6359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:49:22.810806 6381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:22.812039 6359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:22.812189 6359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "29c911b96f084c6b8778a4b052668a5b"
format_stamp: "Formatted at 2026-08-08 04:49:22 on dist-test-slave-2kcd"
server_key: "eade836d2895b827d3390d8093456e09"
server_key_iv: "c34b357515996b2356a668dde35864b8"
server_key_version: "encryptionkey@0"
I20260808 04:49:22.812301 6359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:22.831492 6359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:22.834404 6359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:22.834609 6359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:22.841915 6359 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39119
I20260808 04:49:22.841926 6475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39119 every 8 connection(s)
I20260808 04:49:22.842875 6359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:22.844986 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6359
I20260808 04:49:22.845144 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:22.845474 4964 external_mini_cluster.cc:1428] Setting key c0f4a94702bf920df91327aab96f4423
I20260808 04:49:22.846113 6478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:22.852850 6478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b: Bootstrap starting.
I20260808 04:49:22.854421 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:22.846866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36623 (local address 127.0.0.1:39119)
0808 04:49:22.847303 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:22.847319 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:22.847355 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:49:22.848013 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:22.848029 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:22.848349 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:22.848643 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:22.849026 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:22.849996 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:22.851208 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:22.851653 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:22.851763 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:22.852657 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:22.852661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:22.852674 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:22.853153 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:22.853216 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:22.853240 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:22.853317 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:22.853496 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 04:49:22.853739 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":154,"threads_started":1}
I20260808 04:49:22.855255 6478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:22.856000 6478 log.cc:824] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:22.857429 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:22.858386 6478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b: No bootstrap required, opened a new log
I20260808 04:49:22.863155 6478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c911b96f084c6b8778a4b052668a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260808 04:49:22.863467 6478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:22.863555 6478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c911b96f084c6b8778a4b052668a5b, State: Initialized, Role: FOLLOWER
I20260808 04:49:22.864001 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:22.859734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38867 (local address 127.0.0.1:39119)
0808 04:49:22.859863 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:22.859867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:22.860065 (+ 198us) server_negotiation.cc:407] Connection header received
0808 04:49:22.860173 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:22.860175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:22.860212 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:22.860276 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:22.860576 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:22.861130 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:22.862321 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:22.862555 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:22.862635 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:22.863480 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:22.863487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:22.863491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:22.863510 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:22.863535 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:22.863569 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:22.863666 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:22.863790 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 04:49:22.863840 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:49:22.864108 6478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c911b96f084c6b8778a4b052668a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260808 04:49:22.864276 6478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:22.864347 6478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:22.864456 6478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:22.865489 6478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c911b96f084c6b8778a4b052668a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260808 04:49:22.865785 6478 leader_election.cc:302] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c911b96f084c6b8778a4b052668a5b; no voters:
I20260808 04:49:22.866037 6478 leader_election.cc:288] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:22.866173 6496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:22.866446 6496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [term 1 LEADER]: Becoming Leader. State: Replica: 29c911b96f084c6b8778a4b052668a5b, State: Running, Role: LEADER
I20260808 04:49:22.866778 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c911b96f084c6b8778a4b052668a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260808 04:49:22.867311 6478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:22.868429 6499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c911b96f084c6b8778a4b052668a5b. Latest consensus state: current_term: 1 leader_uuid: "29c911b96f084c6b8778a4b052668a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c911b96f084c6b8778a4b052668a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } } }
I20260808 04:49:22.868579 6499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:22.869024 6516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:22.868973 6498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29c911b96f084c6b8778a4b052668a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c911b96f084c6b8778a4b052668a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } } }
I20260808 04:49:22.869155 6498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:22.871968 6516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:22.877614 6516 catalog_manager.cc:1347] Generated new cluster ID: 41889f158ea443e3a2a7483370738dda
I20260808 04:49:22.877679 6516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:22.897640 6516 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:22.898767 6516 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:22.903995 6516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c911b96f084c6b8778a4b052668a5b: Generated new TSK 0
I20260808 04:49:22.904768 6516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:22.909668 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6359
2026-08-08T04:49:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1664 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:22 dist-test-slave-2kcd krb5kdc[6533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:22 dist-test-slave-2kcd krb5kdc[6533](info): set up 2 sockets
Aug 08 04:49:22 dist-test-slave-2kcd krb5kdc[6533](info): commencing operation
krb5kdc: starting...
W20260808 04:49:24.175400 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:24 dist-test-slave-2kcd krb5kdc[6533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:24.304598 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:24.418565 6557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:24.418865 6557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:24.418939 6557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:24.422879 6557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:24.422967 6557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:24.423019 6557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:24.423079 6557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:24.423120 6557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:24.428258 6557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:24.429364 6557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:24.430406 6557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:24.439132 6571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:24.439211 6557 server_base.cc:1034] running on GCE node
W20260808 04:49:24.439174 6575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:24.439155 6570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:24.439989 6557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:24.441176 6557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:24.442361 6557 hybrid_clock.cc:648] HybridClock initialized: now 1786164564442332 us; error 41 us; skew 500 ppm
Aug 08 04:49:24 dist-test-slave-2kcd krb5kdc[6533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:24.445235 6557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:24.446102 6557 webserver.cc:460] Webserver started at http://127.0.0.1:44365/ using document root <none> and password file <none>
I20260808 04:49:24.446666 6557 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:24.446760 6557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:24.447000 6557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:24.449373 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "190d34a58aab4fe09e5a9eee52a7dfc7"
format_stamp: "Formatted at 2026-08-08 04:49:24 on dist-test-slave-2kcd"
server_key: "26a881bc87c679bde3272a655054501c"
server_key_iv: "bf3328f80b1cbace68dd6341e0a1627c"
server_key_version: "encryptionkey@0"
I20260808 04:49:24.449965 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "190d34a58aab4fe09e5a9eee52a7dfc7"
format_stamp: "Formatted at 2026-08-08 04:49:24 on dist-test-slave-2kcd"
server_key: "26a881bc87c679bde3272a655054501c"
server_key_iv: "bf3328f80b1cbace68dd6341e0a1627c"
server_key_version: "encryptionkey@0"
I20260808 04:49:24.454239 6557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:49:24.456583 6586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:24.457794 6557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:49:24.457939 6557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "190d34a58aab4fe09e5a9eee52a7dfc7"
format_stamp: "Formatted at 2026-08-08 04:49:24 on dist-test-slave-2kcd"
server_key: "26a881bc87c679bde3272a655054501c"
server_key_iv: "bf3328f80b1cbace68dd6341e0a1627c"
server_key_version: "encryptionkey@0"
I20260808 04:49:24.458076 6557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:24.462666 6557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:24.465510 6557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:24.465713 6557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:24.472747 6557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37961
I20260808 04:49:24.472855 6672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37961 every 8 connection(s)
I20260808 04:49:24.473711 6557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:24.476516 6673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:24.481842 6673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7: Bootstrap starting.
I20260808 04:49:24.481947 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6557
I20260808 04:49:24.482105 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:24.482388 4964 external_mini_cluster.cc:1428] Setting key 0c82ab96adec5397c90d004f7a7e7a36
I20260808 04:49:24.484540 6673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:24.485316 6673 log.cc:824] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:24.487468 6673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7: No bootstrap required, opened a new log
I20260808 04:49:24.487694 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:24.483708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58453 (local address 127.0.0.1:37961)
0808 04:49:24.484169 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:24.484184 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:24.484221 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:49:24.484977 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:24.484998 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:24.485287 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:24.485613 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:24.486472 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:24.486477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:24.486488 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:24.486499 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:24.486552 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:24.486571 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:24.486686 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:24.486795 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 04:49:24.486955 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":131,"threads_started":1}
I20260808 04:49:24.490737 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:24.491717 6673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d34a58aab4fe09e5a9eee52a7dfc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } }
I20260808 04:49:24.491955 6673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:24.492018 6673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190d34a58aab4fe09e5a9eee52a7dfc7, State: Initialized, Role: FOLLOWER
I20260808 04:49:24.492455 6673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d34a58aab4fe09e5a9eee52a7dfc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } }
I20260808 04:49:24.492586 6673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:24.492669 6673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:24.492769 6673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:24.494037 6673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d34a58aab4fe09e5a9eee52a7dfc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } }
I20260808 04:49:24.494419 6673 leader_election.cc:302] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 190d34a58aab4fe09e5a9eee52a7dfc7; no voters:
I20260808 04:49:24.494637 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:24.493018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40095 (local address 127.0.0.1:37961)
0808 04:49:24.493149 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:24.493155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:24.493299 (+ 144us) server_negotiation.cc:407] Connection header received
0808 04:49:24.493428 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:24.493430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:24.493457 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:24.493526 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:24.494214 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:24.494218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:24.494220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:24.494233 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:24.494259 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:24.494278 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:24.494348 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:24.494470 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 04:49:24.494501 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:49:24.494817 6673 leader_election.cc:288] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:24.494885 6688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:24.495110 6688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [term 1 LEADER]: Becoming Leader. State: Replica: 190d34a58aab4fe09e5a9eee52a7dfc7, State: Running, Role: LEADER
I20260808 04:49:24.495450 6688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d34a58aab4fe09e5a9eee52a7dfc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } }
I20260808 04:49:24.496061 6673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:24.497196 6690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190d34a58aab4fe09e5a9eee52a7dfc7. Latest consensus state: current_term: 1 leader_uuid: "190d34a58aab4fe09e5a9eee52a7dfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d34a58aab4fe09e5a9eee52a7dfc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } } }
I20260808 04:49:24.497321 6690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:24.497716 6701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:24.497677 6689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "190d34a58aab4fe09e5a9eee52a7dfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d34a58aab4fe09e5a9eee52a7dfc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } } }
I20260808 04:49:24.497812 6689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:24.500761 6701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:24.505872 6701 catalog_manager.cc:1347] Generated new cluster ID: 6ecb5684b0f84d4aa4cb23d76b756d7b
I20260808 04:49:24.505936 6701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:24.515877 6701 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:24.516201 6701 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:24.527446 6701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 190d34a58aab4fe09e5a9eee52a7dfc7: Generated new TSK 0
I20260808 04:49:24.528214 6701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:24.542461 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6557
2026-08-08T04:49:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1624 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:24 dist-test-slave-2kcd krb5kdc[6721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:24 dist-test-slave-2kcd krb5kdc[6721](info): set up 2 sockets
Aug 08 04:49:24 dist-test-slave-2kcd krb5kdc[6721](info): commencing operation
krb5kdc: starting...
W20260808 04:49:25.819002 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:25 dist-test-slave-2kcd krb5kdc[6721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:25.949528 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:26.060029 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:26.060259 6747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:26.060298 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:26.064814 6747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:26.064890 6747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:26.064935 6747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:26.064970 6747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:26.065002 6747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:26.070183 6747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:26.071164 6747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:26.072153 6747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:49:26.080856 6747 server_base.cc:1034] running on GCE node
W20260808 04:49:26.080847 6758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:26.080827 6757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:26.081085 6764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:26.081600 6747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:26.082628 6747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:26.083815 6747 hybrid_clock.cc:648] HybridClock initialized: now 1786164566083803 us; error 25 us; skew 500 ppm
Aug 08 04:49:26 dist-test-slave-2kcd krb5kdc[6721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:26.086561 6747 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:26.087365 6747 webserver.cc:460] Webserver started at http://127.0.0.1:39445/ using document root <none> and password file <none>
I20260808 04:49:26.087841 6747 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:26.087915 6747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:26.088078 6747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:26.090049 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "7de76a4f03014fa796f6c9cba485e951"
format_stamp: "Formatted at 2026-08-08 04:49:26 on dist-test-slave-2kcd"
server_key: "2d0f8e05c3df90c74e8b1f08e21ce24c"
server_key_iv: "f55481e84d01fc4d8f6d2a2f2307c1ab"
server_key_version: "encryptionkey@0"
I20260808 04:49:26.090535 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "7de76a4f03014fa796f6c9cba485e951"
format_stamp: "Formatted at 2026-08-08 04:49:26 on dist-test-slave-2kcd"
server_key: "2d0f8e05c3df90c74e8b1f08e21ce24c"
server_key_iv: "f55481e84d01fc4d8f6d2a2f2307c1ab"
server_key_version: "encryptionkey@0"
I20260808 04:49:26.093991 6747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:49:26.096220 6774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:26.097435 6747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:26.097548 6747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "7de76a4f03014fa796f6c9cba485e951"
format_stamp: "Formatted at 2026-08-08 04:49:26 on dist-test-slave-2kcd"
server_key: "2d0f8e05c3df90c74e8b1f08e21ce24c"
server_key_iv: "f55481e84d01fc4d8f6d2a2f2307c1ab"
server_key_version: "encryptionkey@0"
I20260808 04:49:26.097630 6747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:26.119446 6747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:26.122109 6747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:26.122257 6747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:26.129060 6747 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43477
I20260808 04:49:26.129082 6857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43477 every 8 connection(s)
I20260808 04:49:26.130010 6747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:26.132687 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:26.135752 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6747
I20260808 04:49:26.135927 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:26.136237 4964 external_mini_cluster.cc:1428] Setting key 0725a42fe9f5baed64a13522c836c866
I20260808 04:49:26.138769 6858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951: Bootstrap starting.
I20260808 04:49:26.140933 6858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:26.141649 6858 log.cc:824] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:26.143523 6858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951: No bootstrap required, opened a new log
Aug 08 04:49:26 dist-test-slave-2kcd krb5kdc[6721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:49:26.147535 6858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de76a4f03014fa796f6c9cba485e951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43477 } }
I20260808 04:49:26.147771 6858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:26.147866 6858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de76a4f03014fa796f6c9cba485e951, State: Initialized, Role: FOLLOWER
I20260808 04:49:26.148345 6858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de76a4f03014fa796f6c9cba485e951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43477 } }
I20260808 04:49:26.148505 6858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:26.148612 6858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:26.148716 6858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:26.149861 6858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de76a4f03014fa796f6c9cba485e951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43477 } }
I20260808 04:49:26.150221 6858 leader_election.cc:302] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7de76a4f03014fa796f6c9cba485e951; no voters:
I20260808 04:49:26.150569 6858 leader_election.cc:288] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:26.150655 6865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:26.150899 6865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [term 1 LEADER]: Becoming Leader. State: Replica: 7de76a4f03014fa796f6c9cba485e951, State: Running, Role: LEADER
I20260808 04:49:26.150936 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:26.137792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:40241 (local address 127.0.0.1:43477)
0808 04:49:26.138258 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:26.138274 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:26.138308 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:49:26.138995 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:26.139015 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:26.139339 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:26.139679 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:26.140512 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:26.141632 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:26.142736 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:26.143194 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:26.143377 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:26.145815 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:26.145820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:26.145832 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:26.145860 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:26.147557 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:26.148126 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:26.148132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:26.148137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:26.148245 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:26.148645 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:26.148649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:26.148653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:26.149202 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:26.149421 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:26.149723 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 04:49:26.149959 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":147,"threads_started":1}
I20260808 04:49:26.151280 6865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de76a4f03014fa796f6c9cba485e951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43477 } }
I20260808 04:49:26.151779 6858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:26.152478 6865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de76a4f03014fa796f6c9cba485e951. Latest consensus state: current_term: 1 leader_uuid: "7de76a4f03014fa796f6c9cba485e951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de76a4f03014fa796f6c9cba485e951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43477 } } }
I20260808 04:49:26.152611 6865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:26.153846 6866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7de76a4f03014fa796f6c9cba485e951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de76a4f03014fa796f6c9cba485e951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43477 } } }
I20260808 04:49:26.153959 6866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:26.154362 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 04:49:26.157442 6886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:49:26.157524 6886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:49:26.157594 6871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:26.158427 6871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:26.165019 6871 catalog_manager.cc:1347] Generated new cluster ID: cf0a6fa7ab62438e9abf4acc9d40d91d
I20260808 04:49:26.165078 6871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:26.166224 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:26.156620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:41535 (local address 127.0.0.1:43477)
0808 04:49:26.156722 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:26.156726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:26.157061 (+ 335us) server_negotiation.cc:407] Connection header received
0808 04:49:26.157159 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:26.157162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:26.157210 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:26.157298 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:26.158130 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:26.158896 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:26.160006 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:26.160379 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:26.160497 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:26.162354 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:26.162357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:26.162361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:26.162387 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:26.164127 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:26.164729 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:26.164733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:26.164735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:26.164856 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:26.165297 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:26.165301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:26.165303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:26.165533 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:26.165681 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:26.165996 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 04:49:26.166086 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:49:26.172040 6871 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:26.173197 6871 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:26.187534 6871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7de76a4f03014fa796f6c9cba485e951: Generated new TSK 0
I20260808 04:49:26.188350 6871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:26.201628 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6747
W20260808 04:49:26.215232 6719 connection.cc:535] client connection to 127.0.0.1:43477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43477): Connection reset by peer (error 104)
2026-08-08T04:49:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1660 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:26 dist-test-slave-2kcd krb5kdc[6901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:26 dist-test-slave-2kcd krb5kdc[6901](info): set up 2 sockets
Aug 08 04:49:26 dist-test-slave-2kcd krb5kdc[6901](info): commencing operation
krb5kdc: starting...
W20260808 04:49:27.465440 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:27 dist-test-slave-2kcd krb5kdc[6901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:27.591956 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:27.700953 6923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:27.701236 6923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:27.701323 6923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:27.705250 6923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:27.705330 6923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:27.705368 6923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:27.705427 6923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:27.705466 6923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:27.710721 6923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:27.711704 6923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:27.712670 6923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:27.721021 6929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:27.721029 6933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:27.721035 6928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:27.721306 6923 server_base.cc:1034] running on GCE node
I20260808 04:49:27.721882 6923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:27.722954 6923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:27.724157 6923 hybrid_clock.cc:648] HybridClock initialized: now 1786164567724136 us; error 43 us; skew 500 ppm
Aug 08 04:49:27 dist-test-slave-2kcd krb5kdc[6901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:27.726868 6923 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:27.727734 6923 webserver.cc:460] Webserver started at http://127.0.0.1:40275/ using document root <none> and password file <none>
I20260808 04:49:27.728276 6923 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:27.728366 6923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:27.728616 6923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:27.730654 6923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "e9ad5da145bd488b9526e48fda4d1ca5"
format_stamp: "Formatted at 2026-08-08 04:49:27 on dist-test-slave-2kcd"
server_key: "19676b5e1b43bfe538345dd34a93f65e"
server_key_iv: "b74cfc7cff2d976788cdaaeccec98f92"
server_key_version: "encryptionkey@0"
I20260808 04:49:27.731206 6923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "e9ad5da145bd488b9526e48fda4d1ca5"
format_stamp: "Formatted at 2026-08-08 04:49:27 on dist-test-slave-2kcd"
server_key: "19676b5e1b43bfe538345dd34a93f65e"
server_key_iv: "b74cfc7cff2d976788cdaaeccec98f92"
server_key_version: "encryptionkey@0"
I20260808 04:49:27.734825 6923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 04:49:27.737112 6946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:27.738267 6923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:49:27.738421 6923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "e9ad5da145bd488b9526e48fda4d1ca5"
format_stamp: "Formatted at 2026-08-08 04:49:27 on dist-test-slave-2kcd"
server_key: "19676b5e1b43bfe538345dd34a93f65e"
server_key_iv: "b74cfc7cff2d976788cdaaeccec98f92"
server_key_version: "encryptionkey@0"
I20260808 04:49:27.738530 6923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:27.769758 6923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:27.772379 6923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:27.772586 6923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:27.779683 6923 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34837
I20260808 04:49:27.779716 7038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34837 every 8 connection(s)
I20260808 04:49:27.780576 6923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:27.783298 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:27.788342 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 6923
I20260808 04:49:27.788504 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:27.788817 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5: Bootstrap starting.
I20260808 04:49:27.788905 4964 external_mini_cluster.cc:1428] Setting key 334d4174316995cf121e77f960b9dc74
I20260808 04:49:27.791342 7040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:27.792065 7040 log.cc:824] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:27.794009 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5: No bootstrap required, opened a new log
I20260808 04:49:27.797781 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:27.790247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:45401 (local address 127.0.0.1:34837)
0808 04:49:27.790667 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:27.790681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:27.790716 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:49:27.791330 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:27.791346 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:27.791725 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:27.792042 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:27.792489 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:27.793600 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:27.794544 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:27.795010 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:27.795206 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:27.796100 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:27.796106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:27.796117 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:27.796486 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:27.796538 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:27.796560 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:27.796702 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:27.796927 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 04:49:27.797063 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
I20260808 04:49:27.797720 7040 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ad5da145bd488b9526e48fda4d1ca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260808 04:49:27.797982 7040 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:27.798079 7040 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ad5da145bd488b9526e48fda4d1ca5, State: Initialized, Role: FOLLOWER
I20260808 04:49:27.798527 7040 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ad5da145bd488b9526e48fda4d1ca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260808 04:49:27.798681 7040 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:27.798753 7040 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:27.798849 7040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:27.799787 7040 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ad5da145bd488b9526e48fda4d1ca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260808 04:49:27.800177 7040 leader_election.cc:302] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9ad5da145bd488b9526e48fda4d1ca5; no voters:
I20260808 04:49:27.800545 7040 leader_election.cc:288] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:27.800608 7046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:27.800820 7046 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [term 1 LEADER]: Becoming Leader. State: Replica: e9ad5da145bd488b9526e48fda4d1ca5, State: Running, Role: LEADER
I20260808 04:49:27.800868 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:27.801111 7046 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ad5da145bd488b9526e48fda4d1ca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260808 04:49:27.801827 7040 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:27.802281 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9ad5da145bd488b9526e48fda4d1ca5. Latest consensus state: current_term: 1 leader_uuid: "e9ad5da145bd488b9526e48fda4d1ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ad5da145bd488b9526e48fda4d1ca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } } }
I20260808 04:49:27.802395 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:27.802649 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9ad5da145bd488b9526e48fda4d1ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ad5da145bd488b9526e48fda4d1ca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } } }
I20260808 04:49:27.802810 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:27.802878 7060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:27.806319 7060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:27.808334 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:27.803284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:34023 (local address 127.0.0.1:34837)
0808 04:49:27.803409 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:27.803414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:27.803600 (+ 186us) server_negotiation.cc:407] Connection header received
0808 04:49:27.803820 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:27.803823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:27.803873 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:27.803951 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:27.804402 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:27.805210 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:27.806321 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:27.806644 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:27.806750 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:27.807755 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:27.807758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:27.807761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:27.807774 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:27.807802 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:27.807814 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:27.807873 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:27.808161 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 04:49:27.808198 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":48}
I20260808 04:49:27.812364 7060 catalog_manager.cc:1347] Generated new cluster ID: 5d5ecf46ce2940209fdc06884cb0cb92
I20260808 04:49:27.812431 7060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:27.827749 7060 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:27.829017 7060 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:27.836403 7060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9ad5da145bd488b9526e48fda4d1ca5: Generated new TSK 0
I20260808 04:49:27.837132 7060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:27.844684 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 6923
W20260808 04:49:27.857697 6896 connection.cc:535] client connection to 127.0.0.1:34837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34837): Connection reset by peer (error 104)
2026-08-08T04:49:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1642 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:27 dist-test-slave-2kcd krb5kdc[7091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:27 dist-test-slave-2kcd krb5kdc[7091](info): set up 2 sockets
Aug 08 04:49:27 dist-test-slave-2kcd krb5kdc[7091](info): commencing operation
krb5kdc: starting...
W20260808 04:49:29.101289 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:29 dist-test-slave-2kcd krb5kdc[7091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:29.225765 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:29.336567 7113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:29.336889 7113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:29.336979 7113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:29.340912 7113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:29.340962 7113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:29.340986 7113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:29.341029 7113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:29.341064 7113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:29.346208 7113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:29.347213 7113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:29.348213 7113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:29.356726 7129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:29.356751 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:29.356835 7113 server_base.cc:1034] running on GCE node
W20260808 04:49:29.356952 7128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:29.357535 7113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:29.358639 7113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:29.359810 7113 hybrid_clock.cc:648] HybridClock initialized: now 1786164569359784 us; error 37 us; skew 500 ppm
Aug 08 04:49:29 dist-test-slave-2kcd krb5kdc[7091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:29.362632 7113 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:29.363461 7113 webserver.cc:460] Webserver started at http://127.0.0.1:37471/ using document root <none> and password file <none>
I20260808 04:49:29.363987 7113 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:29.364046 7113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:29.364210 7113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:29.366233 7113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc"
format_stamp: "Formatted at 2026-08-08 04:49:29 on dist-test-slave-2kcd"
server_key: "2fd1474b9ce0fdaa1ad411c9f47667c8"
server_key_iv: "50fc8527db4cf2531d68b4009c38ff95"
server_key_version: "encryptionkey@0"
I20260808 04:49:29.366763 7113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc"
format_stamp: "Formatted at 2026-08-08 04:49:29 on dist-test-slave-2kcd"
server_key: "2fd1474b9ce0fdaa1ad411c9f47667c8"
server_key_iv: "50fc8527db4cf2531d68b4009c38ff95"
server_key_version: "encryptionkey@0"
I20260808 04:49:29.370498 7113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:49:29.372891 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:29.374087 7113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:29.374245 7113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc"
format_stamp: "Formatted at 2026-08-08 04:49:29 on dist-test-slave-2kcd"
server_key: "2fd1474b9ce0fdaa1ad411c9f47667c8"
server_key_iv: "50fc8527db4cf2531d68b4009c38ff95"
server_key_version: "encryptionkey@0"
I20260808 04:49:29.374356 7113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:29.378818 7113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:29.381682 7113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:29.381879 7113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:29.388687 7113 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35905
I20260808 04:49:29.388706 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35905 every 8 connection(s)
I20260808 04:49:29.389570 7113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:29.391538 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 7113
I20260808 04:49:29.391695 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:29.392011 4964 external_mini_cluster.cc:1428] Setting key 05fb6d61b6cad78030fe3be3de5c4de2
I20260808 04:49:29.392409 7229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:29.397454 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:29.393407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:58427 (local address 127.0.0.1:35905)
0808 04:49:29.393857 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:29.393877 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:29.393918 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:49:29.394673 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:29.394696 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:29.395034 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:29.395379 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:29.396106 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:29.396112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:29.396126 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:29.396139 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:29.396200 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:29.396227 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:29.396367 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:29.396437 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 04:49:29.396591 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260808 04:49:29.399580 7229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc: Bootstrap starting.
I20260808 04:49:29.400175 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:29.401860 7229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:29.402737 7229 log.cc:824] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:29.403903 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:29.402303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:46395 (local address 127.0.0.1:35905)
0808 04:49:29.402425 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:29.402429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:29.402625 (+ 196us) server_negotiation.cc:407] Connection header received
0808 04:49:29.402731 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:29.402733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:29.402760 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:29.402821 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:29.403457 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:29.403459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:29.403461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:29.403472 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:29.403513 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:29.403522 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:29.403634 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:29.403714 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 04:49:29.403754 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:49:29.404904 7229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc: No bootstrap required, opened a new log
I20260808 04:49:29.408337 7229 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260808 04:49:29.408582 7229 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:29.408666 7229 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccba07aa17c64d1c94bb4a9269a4b6cc, State: Initialized, Role: FOLLOWER
I20260808 04:49:29.409132 7229 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260808 04:49:29.409315 7229 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:29.409389 7229 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:29.409495 7229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:29.410290 7229 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260808 04:49:29.410573 7229 leader_election.cc:302] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccba07aa17c64d1c94bb4a9269a4b6cc; no voters:
I20260808 04:49:29.410797 7229 leader_election.cc:288] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:29.410931 7242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:29.411206 7242 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [term 1 LEADER]: Becoming Leader. State: Replica: ccba07aa17c64d1c94bb4a9269a4b6cc, State: Running, Role: LEADER
I20260808 04:49:29.411517 7242 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260808 04:49:29.412068 7229 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:29.413230 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccba07aa17c64d1c94bb4a9269a4b6cc. Latest consensus state: current_term: 1 leader_uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260808 04:49:29.413367 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:29.413645 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba07aa17c64d1c94bb4a9269a4b6cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260808 04:49:29.413735 7258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:29.413753 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:29.417275 7258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:29.422564 7258 catalog_manager.cc:1347] Generated new cluster ID: 4826a03a4b384e1083949d9dfa19fe9d
I20260808 04:49:29.422624 7258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:29.433295 7258 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:29.433595 7258 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:29.440272 7258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccba07aa17c64d1c94bb4a9269a4b6cc: Generated new TSK 0
I20260808 04:49:29.441044 7258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:29.448761 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 7113
2026-08-08T04:49:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1600 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:29 dist-test-slave-2kcd krb5kdc[7281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:29 dist-test-slave-2kcd krb5kdc[7281](info): set up 2 sockets
Aug 08 04:49:29 dist-test-slave-2kcd krb5kdc[7281](info): commencing operation
krb5kdc: starting...
W20260808 04:49:30.718766 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:30 dist-test-slave-2kcd krb5kdc[7281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:30.843690 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:30.962392 7299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:30.962666 7299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:30.962708 7299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:30.966626 7299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:30.966678 7299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:30.966697 7299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:30.966714 7299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:30.966729 7299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:30.971968 7299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42175
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:30.973006 7299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:30.974015 7299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:30.982577 7307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:30.982589 7304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:30.982627 7305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:30.982963 7299 server_base.cc:1034] running on GCE node
I20260808 04:49:30.983493 7299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:30.984543 7299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:30.985737 7299 hybrid_clock.cc:648] HybridClock initialized: now 1786164570985710 us; error 46 us; skew 500 ppm
Aug 08 04:49:30 dist-test-slave-2kcd krb5kdc[7281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:30.993067 7299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:30.993952 7299 webserver.cc:460] Webserver started at http://127.0.0.1:43109/ using document root <none> and password file <none>
I20260808 04:49:30.994529 7299 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:30.994623 7299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:30.994870 7299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:30.997048 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "dd179489448e4de5b9f3ecbc89f6d54a"
format_stamp: "Formatted at 2026-08-08 04:49:30 on dist-test-slave-2kcd"
server_key: "7e112f9af457707304f2992f5bc943e9"
server_key_iv: "5c5cb269b9e178675ec26ae0194dc9d1"
server_key_version: "encryptionkey@0"
I20260808 04:49:30.997641 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "dd179489448e4de5b9f3ecbc89f6d54a"
format_stamp: "Formatted at 2026-08-08 04:49:30 on dist-test-slave-2kcd"
server_key: "7e112f9af457707304f2992f5bc943e9"
server_key_iv: "5c5cb269b9e178675ec26ae0194dc9d1"
server_key_version: "encryptionkey@0"
I20260808 04:49:31.001590 7299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:49:31.004256 7317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:31.005607 7299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:31.005766 7299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "dd179489448e4de5b9f3ecbc89f6d54a"
format_stamp: "Formatted at 2026-08-08 04:49:30 on dist-test-slave-2kcd"
server_key: "7e112f9af457707304f2992f5bc943e9"
server_key_iv: "5c5cb269b9e178675ec26ae0194dc9d1"
server_key_version: "encryptionkey@0"
I20260808 04:49:31.005895 7299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:31.029565 7299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:31.032617 7299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:31.032898 7299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:31.040355 7299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42175
I20260808 04:49:31.040385 7410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42175 every 8 connection(s)
I20260808 04:49:31.041324 7299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:31.043213 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 7299
I20260808 04:49:31.043365 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:31.043674 4964 external_mini_cluster.cc:1428] Setting key 543b05b0de7d5a592ed8b30571e369c3
I20260808 04:49:31.044369 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:31.050114 7411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a: Bootstrap starting.
Aug 08 04:49:31 dist-test-slave-2kcd krb5kdc[7281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:49:31.052392 7411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:31.053179 7411 log.cc:824] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:31.055183 7411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a: No bootstrap required, opened a new log
I20260808 04:49:31.057556 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:31.045110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:52235 (local address 127.0.0.1:42175)
0808 04:49:31.045587 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:31.045601 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:31.045637 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:49:31.046282 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:31.046298 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:31.046629 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:31.046960 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:31.047761 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:31.048761 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:31.049662 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:31.050137 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:31.050304 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:31.052880 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:31.052885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:31.052898 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:31.052925 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:31.054612 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:31.055140 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:31.055146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:31.055151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:31.055296 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:31.055657 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:31.055661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:31.055662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:31.056039 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:31.056262 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:31.056594 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 04:49:31.056860 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":162,"threads_started":1}
I20260808 04:49:31.058971 7411 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd179489448e4de5b9f3ecbc89f6d54a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260808 04:49:31.059180 7411 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:31.059242 7411 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd179489448e4de5b9f3ecbc89f6d54a, State: Initialized, Role: FOLLOWER
I20260808 04:49:31.059726 7411 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd179489448e4de5b9f3ecbc89f6d54a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260808 04:49:31.059896 7411 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:31.059988 7411 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:31.060097 7411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:31.060662 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:31.061040 7411 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd179489448e4de5b9f3ecbc89f6d54a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260808 04:49:31.061388 7411 leader_election.cc:302] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd179489448e4de5b9f3ecbc89f6d54a; no voters:
I20260808 04:49:31.061720 7411 leader_election.cc:288] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:31.061796 7422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:31.061993 7422 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [term 1 LEADER]: Becoming Leader. State: Replica: dd179489448e4de5b9f3ecbc89f6d54a, State: Running, Role: LEADER
I20260808 04:49:31.062273 7422 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd179489448e4de5b9f3ecbc89f6d54a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260808 04:49:31.062934 7411 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:31.064206 7425 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd179489448e4de5b9f3ecbc89f6d54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd179489448e4de5b9f3ecbc89f6d54a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } } }
I20260808 04:49:31.064347 7425 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:31.064764 7444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:31.065455 7426 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd179489448e4de5b9f3ecbc89f6d54a. Latest consensus state: current_term: 1 leader_uuid: "dd179489448e4de5b9f3ecbc89f6d54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd179489448e4de5b9f3ecbc89f6d54a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } } }
I20260808 04:49:31.065580 7426 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:31.067909 7444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:31.073345 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:31.062837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:49709 (local address 127.0.0.1:42175)
0808 04:49:31.062961 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:31.062965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:31.063328 (+ 363us) server_negotiation.cc:407] Connection header received
0808 04:49:31.064263 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:31.064266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:31.064310 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:31.064364 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:31.065359 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:31.066144 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:31.067303 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:31.067664 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:31.067792 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:31.069536 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:31.069540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:31.069542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:31.069567 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:31.071146 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:31.071797 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:31.071801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:31.071803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:31.071893 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:31.072373 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:31.072379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:31.072383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:31.072600 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:31.072798 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:31.073089 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 04:49:31.073180 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:49:31.074034 7444 catalog_manager.cc:1347] Generated new cluster ID: 2b2380119ac04b85851101625ada9b19
I20260808 04:49:31.074092 7444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:31.083575 7444 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:31.084391 7444 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:31.096685 7444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd179489448e4de5b9f3ecbc89f6d54a: Generated new TSK 0
I20260808 04:49:31.097344 7444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:31.109709 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 7299
W20260808 04:49:31.125995 7279 connection.cc:535] client connection to 127.0.0.1:42175 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42175): Connection reset by peer (error 104)
2026-08-08T04:49:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1667 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:31 dist-test-slave-2kcd krb5kdc[7465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:31 dist-test-slave-2kcd krb5kdc[7465](info): set up 2 sockets
Aug 08 04:49:31 dist-test-slave-2kcd krb5kdc[7465](info): commencing operation
krb5kdc: starting...
W20260808 04:49:32.394300 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:32 dist-test-slave-2kcd krb5kdc[7465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:32.525288 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:32.640887 7484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:32.641149 7484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:32.641239 7484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:32.645175 7484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:32.645246 7484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:32.645268 7484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:32.645306 7484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:32.645324 7484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:32.650426 7484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:32.651445 7484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:32.652485 7484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:32.661669 7490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:32.661676 7495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:32.661677 7489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:32.661932 7484 server_base.cc:1034] running on GCE node
I20260808 04:49:32.662534 7484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:32.663569 7484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:32.664736 7484 hybrid_clock.cc:648] HybridClock initialized: now 1786164572664719 us; error 29 us; skew 500 ppm
Aug 08 04:49:32 dist-test-slave-2kcd krb5kdc[7465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:32.667601 7484 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:32.668383 7484 webserver.cc:460] Webserver started at http://127.0.0.1:45509/ using document root <none> and password file <none>
I20260808 04:49:32.668960 7484 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:32.669035 7484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:32.669198 7484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:32.671231 7484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "cddc0ff54aab406aa958c08207df82fc"
format_stamp: "Formatted at 2026-08-08 04:49:32 on dist-test-slave-2kcd"
server_key: "0d7c8f0857daf98d9094fc8c791ac09c"
server_key_iv: "e289393a891da4e70e34b3cf9c261d07"
server_key_version: "encryptionkey@0"
I20260808 04:49:32.671746 7484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "cddc0ff54aab406aa958c08207df82fc"
format_stamp: "Formatted at 2026-08-08 04:49:32 on dist-test-slave-2kcd"
server_key: "0d7c8f0857daf98d9094fc8c791ac09c"
server_key_iv: "e289393a891da4e70e34b3cf9c261d07"
server_key_version: "encryptionkey@0"
I20260808 04:49:32.675318 7484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 04:49:32.677654 7506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:32.678817 7484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:32.678948 7484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "cddc0ff54aab406aa958c08207df82fc"
format_stamp: "Formatted at 2026-08-08 04:49:32 on dist-test-slave-2kcd"
server_key: "0d7c8f0857daf98d9094fc8c791ac09c"
server_key_iv: "e289393a891da4e70e34b3cf9c261d07"
server_key_version: "encryptionkey@0"
I20260808 04:49:32.679062 7484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:32.693544 7484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:32.696753 7484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:32.697062 7484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:32.703914 7484 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38407
I20260808 04:49:32.704003 7592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38407 every 8 connection(s)
I20260808 04:49:32.704901 7484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:32.707641 7593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:32.711488 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 7484
I20260808 04:49:32.711629 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:32.711925 4964 external_mini_cluster.cc:1428] Setting key 2756a5227df0d3a7babed6a65330eab6
I20260808 04:49:32.713419 7593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc: Bootstrap starting.
I20260808 04:49:32.715626 7593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:32.716359 7593 log.cc:824] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:32.718169 7593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc: No bootstrap required, opened a new log
I20260808 04:49:32.720760 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:32.713352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:57525 (local address 127.0.0.1:38407)
0808 04:49:32.713768 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:32.713783 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:32.713818 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:49:32.714444 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:32.714459 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:32.714784 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:32.715122 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:32.715534 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:32.716604 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:32.717574 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:32.718041 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:32.718217 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:32.719119 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:32.719124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:32.719137 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:32.719502 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:32.719553 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:32.719576 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:32.719716 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:32.719903 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 04:49:32.720073 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260808 04:49:32.722301 7593 raft_consensus.cc:348] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddc0ff54aab406aa958c08207df82fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } }
I20260808 04:49:32.722570 7593 raft_consensus.cc:374] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:32.722640 7593 raft_consensus.cc:729] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddc0ff54aab406aa958c08207df82fc, State: Initialized, Role: FOLLOWER
I20260808 04:49:32.723107 7593 consensus_queue.cc:260] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddc0ff54aab406aa958c08207df82fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } }
I20260808 04:49:32.723279 7593 raft_consensus.cc:388] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:32.723352 7593 raft_consensus.cc:482] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:32.723451 7593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:32.723901 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:32.724490 7593 raft_consensus.cc:504] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddc0ff54aab406aa958c08207df82fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } }
I20260808 04:49:32.724870 7593 leader_election.cc:302] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cddc0ff54aab406aa958c08207df82fc; no voters:
I20260808 04:49:32.725270 7593 leader_election.cc:288] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:32.725329 7602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:32.725555 7602 raft_consensus.cc:686] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [term 1 LEADER]: Becoming Leader. State: Replica: cddc0ff54aab406aa958c08207df82fc, State: Running, Role: LEADER
I20260808 04:49:32.725898 7602 consensus_queue.cc:237] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddc0ff54aab406aa958c08207df82fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } }
I20260808 04:49:32.726511 7593 sys_catalog.cc:564] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:32.727779 7608 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cddc0ff54aab406aa958c08207df82fc. Latest consensus state: current_term: 1 leader_uuid: "cddc0ff54aab406aa958c08207df82fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddc0ff54aab406aa958c08207df82fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } } }
I20260808 04:49:32.727806 7605 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cddc0ff54aab406aa958c08207df82fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddc0ff54aab406aa958c08207df82fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } } }
I20260808 04:49:32.727952 7608 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:32.727977 7605 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:32.729271 7623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:32.732120 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:32.726375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:51729 (local address 127.0.0.1:38407)
0808 04:49:32.726518 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:32.726522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:32.726537 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:49:32.726668 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:32.726671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:32.726713 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:32.726770 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:32.728340 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:32.729128 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:32.730395 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:32.730628 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:32.730748 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:32.731651 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:32.731655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:32.731658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:32.731670 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:32.731703 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:32.731714 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:32.731813 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:32.731945 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 04:49:32.731979 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:49:32.732455 7623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:32.738148 7623 catalog_manager.cc:1347] Generated new cluster ID: fb8e16fbc54a42df87bd5246e579e2f3
I20260808 04:49:32.738227 7623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:32.763005 7623 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:32.764021 7623 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:32.770974 7623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cddc0ff54aab406aa958c08207df82fc: Generated new TSK 0
I20260808 04:49:32.771659 7623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:32.780879 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 7484
W20260808 04:49:32.801083 7463 connection.cc:535] client connection to 127.0.0.1:38407 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38407): Connection reset by peer (error 104)
2026-08-08T04:49:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1675 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:32 dist-test-slave-2kcd krb5kdc[7640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:32 dist-test-slave-2kcd krb5kdc[7640](info): set up 2 sockets
Aug 08 04:49:32 dist-test-slave-2kcd krb5kdc[7640](info): commencing operation
krb5kdc: starting...
W20260808 04:49:34.042302 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:34 dist-test-slave-2kcd krb5kdc[7640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:49:34.168015 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:34.287575 7664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:34.287935 7664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:34.288041 7664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:34.292223 7664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:34.292310 7664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:34.292351 7664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:34.292407 7664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:34.292446 7664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:34.298113 7664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:34.299220 7664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:34.300283 7664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:34.308876 7677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:34.309043 7664 server_base.cc:1034] running on GCE node
W20260808 04:49:34.308902 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:34.308905 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:34.309825 7664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:34.310953 7664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:34.312156 7664 hybrid_clock.cc:648] HybridClock initialized: now 1786164574312129 us; error 47 us; skew 500 ppm
Aug 08 04:49:34 dist-test-slave-2kcd krb5kdc[7640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:34.315006 7664 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:49:34.315913 7664 webserver.cc:460] Webserver started at http://127.0.0.1:44147/ using document root <none> and password file <none>
I20260808 04:49:34.316615 7664 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:34.316721 7664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:34.317003 7664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:34.319062 7664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "db434db3a0a34343a4ea9994c94b1a1e"
format_stamp: "Formatted at 2026-08-08 04:49:34 on dist-test-slave-2kcd"
server_key: "fc4e10f160f7b8db1e8e6fe564c83090"
server_key_iv: "09977cd67cd1ba4c5c7dec11c4cd0b8d"
server_key_version: "encryptionkey@0"
I20260808 04:49:34.319651 7664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "db434db3a0a34343a4ea9994c94b1a1e"
format_stamp: "Formatted at 2026-08-08 04:49:34 on dist-test-slave-2kcd"
server_key: "fc4e10f160f7b8db1e8e6fe564c83090"
server_key_iv: "09977cd67cd1ba4c5c7dec11c4cd0b8d"
server_key_version: "encryptionkey@0"
I20260808 04:49:34.323458 7664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:49:34.326089 7687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:34.327242 7664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:49:34.327399 7664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "db434db3a0a34343a4ea9994c94b1a1e"
format_stamp: "Formatted at 2026-08-08 04:49:34 on dist-test-slave-2kcd"
server_key: "fc4e10f160f7b8db1e8e6fe564c83090"
server_key_iv: "09977cd67cd1ba4c5c7dec11c4cd0b8d"
server_key_version: "encryptionkey@0"
I20260808 04:49:34.327513 7664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:34.332288 7664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:34.335183 7664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:34.335386 7664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:34.342341 7664 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43853
I20260808 04:49:34.342371 7776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43853 every 8 connection(s)
I20260808 04:49:34.343263 7664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:34.344617 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 7664
I20260808 04:49:34.344823 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:34.345131 4964 external_mini_cluster.cc:1428] Setting key d6643adb4add92f134a445cf4ee21aba
I20260808 04:49:34.346369 7779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:34.350857 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:34.346590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:44161 (local address 127.0.0.1:43853)
0808 04:49:34.347013 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:34.347032 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:34.347074 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:49:34.347919 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:34.347944 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:34.348287 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:34.348662 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:34.349495 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:34.349501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:34.349517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:34.349532 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:34.349594 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:34.349620 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:34.349765 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:34.349833 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 04:49:34.349986 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260808 04:49:34.352962 7779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e: Bootstrap starting.
I20260808 04:49:34.354018 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:34.355345 7779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:34.356159 7779 log.cc:824] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:34.358083 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:34.356263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:55183 (local address 127.0.0.1:43853)
0808 04:49:34.356393 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:34.356396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:34.356633 (+ 237us) server_negotiation.cc:407] Connection header received
0808 04:49:34.356737 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:34.356741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:34.356775 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:34.356906 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:34.357636 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:34.357639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:34.357642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:34.357654 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:34.357683 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:34.357694 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:34.357788 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:34.357913 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 04:49:34.357963 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:49:34.358536 7779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e: No bootstrap required, opened a new log
I20260808 04:49:34.362198 7779 raft_consensus.cc:348] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db434db3a0a34343a4ea9994c94b1a1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } }
I20260808 04:49:34.362498 7779 raft_consensus.cc:374] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:34.362573 7779 raft_consensus.cc:729] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db434db3a0a34343a4ea9994c94b1a1e, State: Initialized, Role: FOLLOWER
I20260808 04:49:34.363091 7779 consensus_queue.cc:260] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db434db3a0a34343a4ea9994c94b1a1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } }
I20260808 04:49:34.363237 7779 raft_consensus.cc:388] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:34.363289 7779 raft_consensus.cc:482] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:34.363370 7779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:34.364244 7779 raft_consensus.cc:504] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db434db3a0a34343a4ea9994c94b1a1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } }
I20260808 04:49:34.364532 7779 leader_election.cc:302] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db434db3a0a34343a4ea9994c94b1a1e; no voters:
I20260808 04:49:34.364862 7779 leader_election.cc:288] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:34.364919 7798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:34.365257 7798 raft_consensus.cc:686] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [term 1 LEADER]: Becoming Leader. State: Replica: db434db3a0a34343a4ea9994c94b1a1e, State: Running, Role: LEADER
I20260808 04:49:34.365602 7798 consensus_queue.cc:237] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db434db3a0a34343a4ea9994c94b1a1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } }
I20260808 04:49:34.366155 7779 sys_catalog.cc:564] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:34.367347 7799 sys_catalog.cc:455] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db434db3a0a34343a4ea9994c94b1a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db434db3a0a34343a4ea9994c94b1a1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } } }
I20260808 04:49:34.367519 7800 sys_catalog.cc:455] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader db434db3a0a34343a4ea9994c94b1a1e. Latest consensus state: current_term: 1 leader_uuid: "db434db3a0a34343a4ea9994c94b1a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db434db3a0a34343a4ea9994c94b1a1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } } }
I20260808 04:49:34.367636 7799 sys_catalog.cc:458] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:34.367944 7800 sys_catalog.cc:458] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:34.368050 7811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:34.370677 7811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:34.376039 7811 catalog_manager.cc:1347] Generated new cluster ID: 84e031c44feb4d73a7bf605bed5a0dd9
I20260808 04:49:34.376144 7811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:34.403168 7811 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:34.403517 7811 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:34.417814 7811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db434db3a0a34343a4ea9994c94b1a1e: Generated new TSK 0
I20260808 04:49:34.418545 7811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:34.425269 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 7664
2026-08-08T04:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1636 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:34 dist-test-slave-2kcd krb5kdc[7828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:34 dist-test-slave-2kcd krb5kdc[7828](info): set up 2 sockets
Aug 08 04:49:34 dist-test-slave-2kcd krb5kdc[7828](info): commencing operation
krb5kdc: starting...
W20260808 04:49:35.687065 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:35 dist-test-slave-2kcd krb5kdc[7828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:35.816308 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:46483
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:35.929944 7850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:35.930225 7850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:35.930301 7850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:35.934304 7850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:35.934383 7850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:35.934418 7850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:35.934473 7850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:35.934514 7850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:35.939702 7850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:46483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:35.940713 7850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:35.941715 7850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:35.950253 7861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:35.950212 7864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:35.950289 7862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:35.950376 7850 server_base.cc:1034] running on GCE node
I20260808 04:49:35.951018 7850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:35.952059 7850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:35.953256 7850 hybrid_clock.cc:648] HybridClock initialized: now 1786164575953230 us; error 37 us; skew 500 ppm
Aug 08 04:49:35 dist-test-slave-2kcd krb5kdc[7828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164575, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:35.955900 7850 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:35.956681 7850 webserver.cc:460] Webserver started at http://127.4.217.62:41829/ using document root <none> and password file <none>
I20260808 04:49:35.957245 7850 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:35.957306 7850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:35.957461 7850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:35.959287 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "bd7cd5a5a6124691921ec2e4c0100737"
format_stamp: "Formatted at 2026-08-08 04:49:35 on dist-test-slave-2kcd"
server_key: "36ef51051fe0a9bbab0016ead3974e51"
server_key_iv: "fca0fbd0438e0c62c0ac3c2f53b25c3c"
server_key_version: "encryptionkey@0"
I20260808 04:49:35.959781 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "bd7cd5a5a6124691921ec2e4c0100737"
format_stamp: "Formatted at 2026-08-08 04:49:35 on dist-test-slave-2kcd"
server_key: "36ef51051fe0a9bbab0016ead3974e51"
server_key_iv: "fca0fbd0438e0c62c0ac3c2f53b25c3c"
server_key_version: "encryptionkey@0"
I20260808 04:49:35.963351 7850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:49:35.965968 7877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:35.967322 7850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:49:35.967463 7850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "bd7cd5a5a6124691921ec2e4c0100737"
format_stamp: "Formatted at 2026-08-08 04:49:35 on dist-test-slave-2kcd"
server_key: "36ef51051fe0a9bbab0016ead3974e51"
server_key_iv: "fca0fbd0438e0c62c0ac3c2f53b25c3c"
server_key_version: "encryptionkey@0"
I20260808 04:49:35.967595 7850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:35.999071 7850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:36.001901 7850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:36.002136 7850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:36.009148 7850 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:46483
I20260808 04:49:36.009162 7965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:46483 every 8 connection(s)
I20260808 04:49:36.010087 7850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:36.012846 7966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:36.013844 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 7850
I20260808 04:49:36.014014 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:36.014313 4964 external_mini_cluster.cc:1428] Setting key 1cc57b2f35ca8391812a3cc0f9bd647b
I20260808 04:49:36.018872 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737: Bootstrap starting.
I20260808 04:49:36.021188 7966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:36.021973 7966 log.cc:824] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:49:36 dist-test-slave-2kcd krb5kdc[7828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:36.024015 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737: No bootstrap required, opened a new log
I20260808 04:49:36.027587 7966 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7cd5a5a6124691921ec2e4c0100737" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46483 } }
I20260808 04:49:36.027905 7966 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:36.027979 7966 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7cd5a5a6124691921ec2e4c0100737, State: Initialized, Role: FOLLOWER
I20260808 04:49:36.028317 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:36.015695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43171 (local address 127.4.217.62:46483)
0808 04:49:36.016146 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:36.016161 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:36.016196 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:49:36.016882 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:36.016900 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:36.017227 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:36.017547 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:36.018368 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:36.019369 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:36.020234 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:36.020698 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:36.020914 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:36.023430 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:36.023436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:36.023448 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:36.023475 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:36.025287 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:36.025812 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:36.025817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:36.025821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:36.025927 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:36.026281 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:36.026284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:36.026286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:36.026650 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:36.026883 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:36.027231 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 04:49:36.027443 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":140,"threads_started":1}
I20260808 04:49:36.028460 7966 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7cd5a5a6124691921ec2e4c0100737" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46483 } }
I20260808 04:49:36.028635 7966 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:36.028707 7966 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:36.028828 7966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:36.029847 7966 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7cd5a5a6124691921ec2e4c0100737" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46483 } }
I20260808 04:49:36.030198 7966 leader_election.cc:302] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd7cd5a5a6124691921ec2e4c0100737; no voters:
I20260808 04:49:36.030579 7966 leader_election.cc:288] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:36.030704 7972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:36.030956 7972 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [term 1 LEADER]: Becoming Leader. State: Replica: bd7cd5a5a6124691921ec2e4c0100737, State: Running, Role: LEADER
I20260808 04:49:36.031329 7972 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7cd5a5a6124691921ec2e4c0100737" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46483 } }
I20260808 04:49:36.031432 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:36.031876 7966 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:36.032375 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd7cd5a5a6124691921ec2e4c0100737. Latest consensus state: current_term: 1 leader_uuid: "bd7cd5a5a6124691921ec2e4c0100737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7cd5a5a6124691921ec2e4c0100737" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46483 } } }
I20260808 04:49:36.032488 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:36.032855 7974 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd7cd5a5a6124691921ec2e4c0100737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7cd5a5a6124691921ec2e4c0100737" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 46483 } } }
I20260808 04:49:36.032945 7974 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:36.033071 7977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:36.036221 7977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:36.041967 7977 catalog_manager.cc:1347] Generated new cluster ID: 76b7a4dc647a4f6897d84fa882014789
I20260808 04:49:36.042045 7977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:36.043627 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:36.033908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53091 (local address 127.4.217.62:46483)
0808 04:49:36.034035 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:36.034040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:36.034987 (+ 947us) server_negotiation.cc:407] Connection header received
0808 04:49:36.035109 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:36.035113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:36.035161 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:36.035245 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:36.036014 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:36.036805 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:36.038008 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:36.038359 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:36.038437 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:36.040271 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:36.040276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:36.040278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:36.040304 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:36.041778 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:36.042338 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:36.042341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:36.042343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:36.042423 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:36.042769 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:36.042772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:36.042774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:36.042939 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:36.043121 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:36.043386 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 04:49:36.043480 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 04:49:36.058071 7977 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:36.059255 7977 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:36.066963 7977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd7cd5a5a6124691921ec2e4c0100737: Generated new TSK 0
I20260808 04:49:36.067706 7977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:36.073194 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 7850
W20260808 04:49:36.091120 7826 connection.cc:535] client connection to 127.4.217.62:46483 recv error: Network error: failed to read from TLS socket (remote: 127.4.217.62:46483): Connection reset by peer (error 104)
2026-08-08T04:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1653 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:36 dist-test-slave-2kcd krb5kdc[8011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:36 dist-test-slave-2kcd krb5kdc[8011](info): set up 2 sockets
Aug 08 04:49:36 dist-test-slave-2kcd krb5kdc[8011](info): commencing operation
krb5kdc: starting...
W20260808 04:49:37.344090 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:37 dist-test-slave-2kcd krb5kdc[8011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:37.473938 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:39817
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:37.585654 8036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:37.585958 8036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:37.586041 8036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:37.589977 8036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:37.590067 8036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:37.590108 8036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:37.590166 8036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:37.590206 8036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:37.595369 8036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:39817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:39817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:37.596428 8036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:37.597462 8036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:49:37.605966 8036 server_base.cc:1034] running on GCE node
W20260808 04:49:37.606015 8049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:37.605980 8045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:37.606163 8052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:37.606814 8036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:37.607898 8036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:37.609058 8036 hybrid_clock.cc:648] HybridClock initialized: now 1786164577609041 us; error 33 us; skew 500 ppm
Aug 08 04:49:37 dist-test-slave-2kcd krb5kdc[8011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164577, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:37.611825 8036 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:37.612669 8036 webserver.cc:460] Webserver started at http://127.4.217.62:44479/ using document root <none> and password file <none>
I20260808 04:49:37.613269 8036 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:37.613365 8036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:37.613597 8036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:37.615787 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "f56c8c2df1e542ee8003449c6863fa8b"
format_stamp: "Formatted at 2026-08-08 04:49:37 on dist-test-slave-2kcd"
server_key: "d414e00f9981d6b58b5f5c66fc677095"
server_key_iv: "5c305c299898c4f00f88533e98b2565a"
server_key_version: "encryptionkey@0"
I20260808 04:49:37.616483 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "f56c8c2df1e542ee8003449c6863fa8b"
format_stamp: "Formatted at 2026-08-08 04:49:37 on dist-test-slave-2kcd"
server_key: "d414e00f9981d6b58b5f5c66fc677095"
server_key_iv: "5c305c299898c4f00f88533e98b2565a"
server_key_version: "encryptionkey@0"
I20260808 04:49:37.620997 8036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260808 04:49:37.623421 8062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:37.624605 8036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:37.624735 8036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "f56c8c2df1e542ee8003449c6863fa8b"
format_stamp: "Formatted at 2026-08-08 04:49:37 on dist-test-slave-2kcd"
server_key: "d414e00f9981d6b58b5f5c66fc677095"
server_key_iv: "5c305c299898c4f00f88533e98b2565a"
server_key_version: "encryptionkey@0"
I20260808 04:49:37.624886 8036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:37.640268 8036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:37.643442 8036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:37.643649 8036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:37.650954 8036 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:39817
I20260808 04:49:37.650985 8149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:39817 every 8 connection(s)
I20260808 04:49:37.651899 8036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:37.654733 8150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:37.660293 8150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b: Bootstrap starting.
I20260808 04:49:37.660415 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 8036
I20260808 04:49:37.660630 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:37.660954 4964 external_mini_cluster.cc:1428] Setting key fe3eca25b3abfc9fa175764cd64d5abf
I20260808 04:49:37.663221 8150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:37.664124 8150 log.cc:824] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:37.666501 8150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b: No bootstrap required, opened a new log
I20260808 04:49:37.670964 8150 raft_consensus.cc:348] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56c8c2df1e542ee8003449c6863fa8b" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39817 } }
I20260808 04:49:37.671226 8150 raft_consensus.cc:374] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:37.671285 8150 raft_consensus.cc:729] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56c8c2df1e542ee8003449c6863fa8b, State: Initialized, Role: FOLLOWER
I20260808 04:49:37.671551 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:37.662366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47521 (local address 127.4.217.62:39817)
0808 04:49:37.662803 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:37.662827 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:37.662870 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:49:37.663750 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:37.663774 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:37.664222 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:37.664589 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:37.665098 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:37.666505 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:37.667521 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:37.668223 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:37.668401 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:37.669464 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:37.669471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:37.669490 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:37.670025 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:37.670093 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:37.670122 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:37.670262 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:37.670459 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 04:49:37.670599 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":119,"threads_started":1}
I20260808 04:49:37.671780 8150 consensus_queue.cc:260] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56c8c2df1e542ee8003449c6863fa8b" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39817 } }
I20260808 04:49:37.671957 8150 raft_consensus.cc:388] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:37.672025 8150 raft_consensus.cc:482] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:37.672125 8150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:37.673146 8150 raft_consensus.cc:504] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56c8c2df1e542ee8003449c6863fa8b" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39817 } }
I20260808 04:49:37.673504 8150 leader_election.cc:302] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f56c8c2df1e542ee8003449c6863fa8b; no voters:
I20260808 04:49:37.673887 8150 leader_election.cc:288] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:37.673996 8160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:37.674202 8160 raft_consensus.cc:686] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [term 1 LEADER]: Becoming Leader. State: Replica: f56c8c2df1e542ee8003449c6863fa8b, State: Running, Role: LEADER
I20260808 04:49:37.674597 8160 consensus_queue.cc:237] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56c8c2df1e542ee8003449c6863fa8b" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39817 } }
I20260808 04:49:37.674782 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:37.675199 8150 sys_catalog.cc:564] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:37.676322 8163 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f56c8c2df1e542ee8003449c6863fa8b. Latest consensus state: current_term: 1 leader_uuid: "f56c8c2df1e542ee8003449c6863fa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56c8c2df1e542ee8003449c6863fa8b" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39817 } } }
I20260808 04:49:37.676472 8163 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:37.676817 8161 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f56c8c2df1e542ee8003449c6863fa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56c8c2df1e542ee8003449c6863fa8b" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 39817 } } }
I20260808 04:49:37.676918 8161 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:37.677237 8175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:37.680655 8175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:37.682324 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:37.677457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49063 (local address 127.4.217.62:39817)
0808 04:49:37.677566 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:37.677570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:37.678102 (+ 532us) server_negotiation.cc:407] Connection header received
0808 04:49:37.678205 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:37.678207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:37.678243 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:37.678305 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:37.678603 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:37.679282 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:37.680391 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:37.680719 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:37.680854 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:37.681806 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:37.681810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:37.681813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:37.681830 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:37.681860 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:37.681874 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:37.681981 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:37.682154 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 04:49:37.682190 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:49:37.686758 8175 catalog_manager.cc:1347] Generated new cluster ID: e3b4740a56734537a869f329a0efac86
I20260808 04:49:37.686820 8175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:37.715056 8175 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:37.716323 8175 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:37.724716 8175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f56c8c2df1e542ee8003449c6863fa8b: Generated new TSK 0
I20260808 04:49:37.725510 8175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:37.729723 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 8036
W20260808 04:49:37.746184 8007 connection.cc:535] client connection to 127.4.217.62:39817 recv error: Network error: failed to read from TLS socket (remote: 127.4.217.62:39817): Connection reset by peer (error 104)
2026-08-08T04:49:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1655 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:37 dist-test-slave-2kcd krb5kdc[8199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:37 dist-test-slave-2kcd krb5kdc[8199](info): set up 2 sockets
Aug 08 04:49:37 dist-test-slave-2kcd krb5kdc[8199](info): commencing operation
krb5kdc: starting...
W20260808 04:49:38.996165 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:39 dist-test-slave-2kcd krb5kdc[8199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:39.127866 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:36917
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:36917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:39.242411 8224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:39.242718 8224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:39.242810 8224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:39.246779 8224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:39.246869 8224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:39.246907 8224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:39.246942 8224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:39.246984 8224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:39.252259 8224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:36917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:36917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:39.253363 8224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:39.254382 8224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:39.263025 8232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:39.263136 8237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:39.263131 8224 server_base.cc:1034] running on GCE node
W20260808 04:49:39.263226 8234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:39.263921 8224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:39.265046 8224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:39.266232 8224 hybrid_clock.cc:648] HybridClock initialized: now 1786164579266202 us; error 43 us; skew 500 ppm
Aug 08 04:49:39 dist-test-slave-2kcd krb5kdc[8199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164579, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:39.268896 8224 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:39.269717 8224 webserver.cc:460] Webserver started at http://127.4.217.62:39843/ using document root <none> and password file <none>
I20260808 04:49:39.270267 8224 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:39.270362 8224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:39.270608 8224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:39.272606 8224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "2660297c032d48e18d9f1d6a47f46410"
format_stamp: "Formatted at 2026-08-08 04:49:39 on dist-test-slave-2kcd"
server_key: "b15c6f3b80d5911feeae99c34c3e87b2"
server_key_iv: "b7129c12245258a79a60292dd78ec589"
server_key_version: "encryptionkey@0"
I20260808 04:49:39.273231 8224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "2660297c032d48e18d9f1d6a47f46410"
format_stamp: "Formatted at 2026-08-08 04:49:39 on dist-test-slave-2kcd"
server_key: "b15c6f3b80d5911feeae99c34c3e87b2"
server_key_iv: "b7129c12245258a79a60292dd78ec589"
server_key_version: "encryptionkey@0"
I20260808 04:49:39.276993 8224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:49:39.279240 8245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:39.280351 8224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:39.280499 8224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "2660297c032d48e18d9f1d6a47f46410"
format_stamp: "Formatted at 2026-08-08 04:49:39 on dist-test-slave-2kcd"
server_key: "b15c6f3b80d5911feeae99c34c3e87b2"
server_key_iv: "b7129c12245258a79a60292dd78ec589"
server_key_version: "encryptionkey@0"
I20260808 04:49:39.280608 8224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:39.285138 8224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:39.287791 8224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:39.287989 8224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:39.294725 8224 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:36917
I20260808 04:49:39.294785 8326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:36917 every 8 connection(s)
I20260808 04:49:39.295627 8224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:39.298280 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:39.303529 8327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410: Bootstrap starting.
I20260808 04:49:39.304579 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 8224
I20260808 04:49:39.304741 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:39.305055 4964 external_mini_cluster.cc:1428] Setting key 9b764511aaffbb35c484b3e96614ad98
I20260808 04:49:39.306056 8327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:39.306819 8327 log.cc:824] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:39.308987 8327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410: No bootstrap required, opened a new log
I20260808 04:49:39.309990 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:39.306128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57493 (local address 127.4.217.62:36917)
0808 04:49:39.306604 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:39.306618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:39.306650 (+ 32us) server_negotiation.cc:407] Connection header received
0808 04:49:39.307291 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:39.307309 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:39.307574 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:39.307832 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:39.308703 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:39.308707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:39.308718 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:39.308729 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:39.308818 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:39.308853 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:39.308939 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:39.309161 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 04:49:39.309327 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":145,"threads_started":1}
I20260808 04:49:39.312881 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:39.312965 8327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660297c032d48e18d9f1d6a47f46410" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36917 } }
I20260808 04:49:39.313205 8327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:39.313254 8327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2660297c032d48e18d9f1d6a47f46410, State: Initialized, Role: FOLLOWER
I20260808 04:49:39.313709 8327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660297c032d48e18d9f1d6a47f46410" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36917 } }
I20260808 04:49:39.313879 8327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:39.313949 8327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:39.314043 8327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:39.315187 8327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660297c032d48e18d9f1d6a47f46410" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36917 } }
I20260808 04:49:39.315582 8327 leader_election.cc:302] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2660297c032d48e18d9f1d6a47f46410; no voters:
I20260808 04:49:39.315920 8327 leader_election.cc:288] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:39.316023 8347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:39.316308 8347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [term 1 LEADER]: Becoming Leader. State: Replica: 2660297c032d48e18d9f1d6a47f46410, State: Running, Role: LEADER
I20260808 04:49:39.316661 8347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660297c032d48e18d9f1d6a47f46410" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36917 } }
I20260808 04:49:39.316954 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:39.315200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37097 (local address 127.4.217.62:36917)
0808 04:49:39.315384 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:39.315388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:39.315463 (+ 75us) server_negotiation.cc:407] Connection header received
0808 04:49:39.315590 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:39.315592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:39.315617 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:39.315677 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:39.316395 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:39.316398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:39.316401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:39.316412 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:39.316437 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:39.316450 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:39.316522 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:39.316811 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 04:49:39.316848 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 04:49:39.317227 8327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:39.317961 8348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2660297c032d48e18d9f1d6a47f46410. Latest consensus state: current_term: 1 leader_uuid: "2660297c032d48e18d9f1d6a47f46410" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660297c032d48e18d9f1d6a47f46410" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36917 } } }
I20260808 04:49:39.318092 8348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:39.318500 8358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:39.318766 8350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2660297c032d48e18d9f1d6a47f46410" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660297c032d48e18d9f1d6a47f46410" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36917 } } }
I20260808 04:49:39.318924 8350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:39.321815 8358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:39.326872 8358 catalog_manager.cc:1347] Generated new cluster ID: 308cf8b1b2ba48e887c64a4aec48b6f4
I20260808 04:49:39.326939 8358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:39.337857 8358 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:39.338188 8358 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:39.346081 8358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2660297c032d48e18d9f1d6a47f46410: Generated new TSK 0
I20260808 04:49:39.346779 8358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:39.352114 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 8224
2026-08-08T04:49:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1627 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:39 dist-test-slave-2kcd krb5kdc[8376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:39 dist-test-slave-2kcd krb5kdc[8376](info): set up 2 sockets
Aug 08 04:49:39 dist-test-slave-2kcd krb5kdc[8376](info): commencing operation
krb5kdc: starting...
W20260808 04:49:40.626677 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:40 dist-test-slave-2kcd krb5kdc[8376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:40.762199 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41795
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:40.873127 8393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:40.873399 8393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:40.873492 8393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:40.877475 8393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:40.877553 8393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:40.877590 8393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:40.877646 8393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:40.877682 8393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:40.882812 8393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:40.883858 8393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:40.884917 8393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:40.893302 8403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:40.893347 8402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:40.893539 8393 server_base.cc:1034] running on GCE node
W20260808 04:49:40.893337 8406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:40.894191 8393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:40.895280 8393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:40.896446 8393 hybrid_clock.cc:648] HybridClock initialized: now 1786164580896417 us; error 42 us; skew 500 ppm
Aug 08 04:49:40 dist-test-slave-2kcd krb5kdc[8376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:40.899294 8393 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:40.900187 8393 webserver.cc:460] Webserver started at http://127.4.217.62:44939/ using document root <none> and password file <none>
I20260808 04:49:40.900745 8393 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:40.900895 8393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:40.901142 8393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:40.903126 8393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7"
format_stamp: "Formatted at 2026-08-08 04:49:40 on dist-test-slave-2kcd"
server_key: "4eb970a3582d38c0aae736a252190ab1"
server_key_iv: "34dc189afdc3b4d79d68d5cd6238df36"
server_key_version: "encryptionkey@0"
I20260808 04:49:40.903699 8393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7"
format_stamp: "Formatted at 2026-08-08 04:49:40 on dist-test-slave-2kcd"
server_key: "4eb970a3582d38c0aae736a252190ab1"
server_key_iv: "34dc189afdc3b4d79d68d5cd6238df36"
server_key_version: "encryptionkey@0"
I20260808 04:49:40.907932 8393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:49:40.911433 8428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:40.913070 8393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 04:49:40.913272 8393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7"
format_stamp: "Formatted at 2026-08-08 04:49:40 on dist-test-slave-2kcd"
server_key: "4eb970a3582d38c0aae736a252190ab1"
server_key_iv: "34dc189afdc3b4d79d68d5cd6238df36"
server_key_version: "encryptionkey@0"
I20260808 04:49:40.913497 8393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:40.927821 8393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:40.930951 8393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:40.931149 8393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:40.938247 8393 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:41795
I20260808 04:49:40.938270 8508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:41795 every 8 connection(s)
I20260808 04:49:40.939175 8393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:40.942407 8510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:40.947853 8510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7: Bootstrap starting.
I20260808 04:49:40.949280 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 8393
I20260808 04:49:40.949445 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:40.949735 4964 external_mini_cluster.cc:1428] Setting key 64935a89720712ea80cd1c887833209b
I20260808 04:49:40.950726 8510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:40.951567 8510 log.cc:824] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:40.953830 8510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7: No bootstrap required, opened a new log
I20260808 04:49:40.958213 8510 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41795 } }
I20260808 04:49:40.958516 8510 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:40.958609 8510 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6bc203cb3a40eb9c4c5453e817c6c7, State: Initialized, Role: FOLLOWER
Aug 08 04:49:40 dist-test-slave-2kcd krb5kdc[8376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:40.959120 8510 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41795 } }
I20260808 04:49:40.959295 8510 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:40.959367 8510 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:40.959462 8510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:40.960381 8510 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41795 } }
I20260808 04:49:40.960741 8510 leader_election.cc:302] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec6bc203cb3a40eb9c4c5453e817c6c7; no voters:
I20260808 04:49:40.961155 8510 leader_election.cc:288] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:40.961241 8515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:40.961524 8515 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [term 1 LEADER]: Becoming Leader. State: Replica: ec6bc203cb3a40eb9c4c5453e817c6c7, State: Running, Role: LEADER
I20260808 04:49:40.961831 8515 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41795 } }
I20260808 04:49:40.962527 8510 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:40.964475 8516 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41795 } } }
I20260808 04:49:40.964617 8516 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:40.964475 8518 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec6bc203cb3a40eb9c4c5453e817c6c7. Latest consensus state: current_term: 1 leader_uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6bc203cb3a40eb9c4c5453e817c6c7" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41795 } } }
I20260808 04:49:40.964959 8518 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:40.965539 8540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:40.967049 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:40.950879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55037 (local address 127.4.217.62:41795)
0808 04:49:40.951337 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:40.951352 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:40.951386 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:49:40.952191 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:40.952215 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:40.952591 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:40.952936 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:40.953820 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:40.954824 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:40.956064 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:40.956550 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:40.956667 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:40.959790 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:40.959795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:40.959809 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:40.959838 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:40.961985 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:40.962553 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:40.962560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:40.962566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:40.962715 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:40.963100 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:40.963103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:40.963105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:40.963628 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:40.965406 (+ 1778us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:40.965615 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 04:49:40.965994 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":139,"threads_started":1}
I20260808 04:49:40.968887 8540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:40.970562 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:40.975268 8540 catalog_manager.cc:1347] Generated new cluster ID: 4c35bf0e2f194dfc9ebb2dd6ea36b67c
I20260808 04:49:40.975332 8540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:40.982358 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:40.973277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48891 (local address 127.4.217.62:41795)
0808 04:49:40.973404 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:40.973408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:40.973648 (+ 240us) server_negotiation.cc:407] Connection header received
0808 04:49:40.973851 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:40.973855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:40.973917 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:40.974009 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:40.974857 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:40.975661 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:40.976845 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:40.977217 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:40.977332 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:40.978942 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:40.978946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:40.978949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:40.978977 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:40.980655 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:40.981139 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:40.981143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:40.981145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:40.981233 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:40.981503 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:40.981507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:40.981508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:40.981722 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:40.981851 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:40.982081 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 04:49:40.982198 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:49:40.997008 8540 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:40.998366 8540 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:41.012879 8540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec6bc203cb3a40eb9c4c5453e817c6c7: Generated new TSK 0
I20260808 04:49:41.013655 8540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:41.029946 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 8393
2026-08-08T04:49:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1672 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:41 dist-test-slave-2kcd krb5kdc[8554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:41 dist-test-slave-2kcd krb5kdc[8554](info): set up 2 sockets
Aug 08 04:49:41 dist-test-slave-2kcd krb5kdc[8554](info): commencing operation
krb5kdc: starting...
W20260808 04:49:42.293058 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:42 dist-test-slave-2kcd krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:42.426556 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:36263
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:32903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:36263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:42.553623 8573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:42.553913 8573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:42.553990 8573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:42.558205 8573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:42.558295 8573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:42.558331 8573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:42.558385 8573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:42.558422 8573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:42.564095 8573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:32903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:36263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:36263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:42.565402 8573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:42.566553 8573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:42.575399 8588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:42.575414 8591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:42.575464 8586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:42.575668 8573 server_base.cc:1034] running on GCE node
I20260808 04:49:42.576329 8573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:42.577452 8573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:42.578624 8573 hybrid_clock.cc:648] HybridClock initialized: now 1786164582578596 us; error 39 us; skew 500 ppm
Aug 08 04:49:42 dist-test-slave-2kcd krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:42.581444 8573 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:42.582284 8573 webserver.cc:460] Webserver started at http://127.4.217.62:38885/ using document root <none> and password file <none>
I20260808 04:49:42.582841 8573 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:42.582902 8573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:42.583143 8573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:42.585229 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "d8902e1b4b1c4019b86771f8d0e9512f"
format_stamp: "Formatted at 2026-08-08 04:49:42 on dist-test-slave-2kcd"
server_key: "4887a88d48827a03b50eb7780e9ba3f9"
server_key_iv: "97df08b8996591bca104ca99556a977c"
server_key_version: "encryptionkey@0"
I20260808 04:49:42.585803 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "d8902e1b4b1c4019b86771f8d0e9512f"
format_stamp: "Formatted at 2026-08-08 04:49:42 on dist-test-slave-2kcd"
server_key: "4887a88d48827a03b50eb7780e9ba3f9"
server_key_iv: "97df08b8996591bca104ca99556a977c"
server_key_version: "encryptionkey@0"
I20260808 04:49:42.589455 8573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:49:42.591750 8600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:42.592974 8573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260808 04:49:42.593109 8573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "d8902e1b4b1c4019b86771f8d0e9512f"
format_stamp: "Formatted at 2026-08-08 04:49:42 on dist-test-slave-2kcd"
server_key: "4887a88d48827a03b50eb7780e9ba3f9"
server_key_iv: "97df08b8996591bca104ca99556a977c"
server_key_version: "encryptionkey@0"
I20260808 04:49:42.593218 8573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:42.609217 8573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:42.611843 8573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:42.612052 8573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:42.618856 8573 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:36263
I20260808 04:49:42.618927 8680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:36263 every 8 connection(s)
I20260808 04:49:42.619791 8573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:42.622892 8681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:42.625324 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 8573
I20260808 04:49:42.625490 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:42.625772 4964 external_mini_cluster.cc:1428] Setting key 62ad82a762a850299f249d5224b189d3
I20260808 04:49:42.630064 8681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f: Bootstrap starting.
I20260808 04:49:42.632714 8681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:42.633514 8681 log.cc:824] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:42.634632 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:42.627234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37383 (local address 127.4.217.62:36263)
0808 04:49:42.627639 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:42.627654 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:42.627689 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:49:42.628295 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:42.628311 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:42.628638 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:42.628960 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:42.629359 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:42.630372 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:42.631613 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:42.632058 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:42.632166 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:42.633038 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:42.633043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:42.633055 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:42.633407 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:42.633458 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:42.633479 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:42.633555 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:42.633751 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 04:49:42.633981 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":129,"threads_started":1}
I20260808 04:49:42.635807 8681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f: No bootstrap required, opened a new log
I20260808 04:49:42.638443 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:42.640089 8681 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8902e1b4b1c4019b86771f8d0e9512f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36263 } }
I20260808 04:49:42.640327 8681 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:42.640385 8681 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8902e1b4b1c4019b86771f8d0e9512f, State: Initialized, Role: FOLLOWER
I20260808 04:49:42.641088 8681 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8902e1b4b1c4019b86771f8d0e9512f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36263 } }
I20260808 04:49:42.641301 8681 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:42.641392 8681 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:42.641536 8681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:42.642867 8681 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8902e1b4b1c4019b86771f8d0e9512f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36263 } }
I20260808 04:49:42.643265 8681 leader_election.cc:302] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8902e1b4b1c4019b86771f8d0e9512f; no voters:
I20260808 04:49:42.643692 8681 leader_election.cc:288] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:42.643774 8696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:42.644063 8696 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [term 1 LEADER]: Becoming Leader. State: Replica: d8902e1b4b1c4019b86771f8d0e9512f, State: Running, Role: LEADER
I20260808 04:49:42.644423 8696 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8902e1b4b1c4019b86771f8d0e9512f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36263 } }
I20260808 04:49:42.645172 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:42.640450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35881 (local address 127.4.217.62:36263)
0808 04:49:42.640570 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:42.640575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:42.640821 (+ 246us) server_negotiation.cc:407] Connection header received
0808 04:49:42.640909 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:42.640911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:42.640948 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:42.641016 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:42.641501 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:42.642062 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:42.643291 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:42.643600 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:42.643720 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:42.644644 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:42.644648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:42.644651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:42.644663 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:42.644689 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:42.644701 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:42.644746 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:42.644953 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 04:49:42.645014 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:49:42.645520 8681 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:42.646413 8698 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8902e1b4b1c4019b86771f8d0e9512f. Latest consensus state: current_term: 1 leader_uuid: "d8902e1b4b1c4019b86771f8d0e9512f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8902e1b4b1c4019b86771f8d0e9512f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36263 } } }
I20260808 04:49:42.646442 8699 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8902e1b4b1c4019b86771f8d0e9512f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8902e1b4b1c4019b86771f8d0e9512f" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 36263 } } }
I20260808 04:49:42.646588 8698 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:42.646588 8699 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:42.647045 8711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:42.650341 8711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:42.656494 8711 catalog_manager.cc:1347] Generated new cluster ID: 627707e4239f4fb2a3fec8fa6792a64c
I20260808 04:49:42.656569 8711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:42.666575 8711 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:42.667456 8711 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:42.676676 8711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8902e1b4b1c4019b86771f8d0e9512f: Generated new TSK 0
I20260808 04:49:42.677335 8711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:42.681012 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 8573
2026-08-08T04:49:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1646 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:42 dist-test-slave-2kcd krb5kdc[8733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:42 dist-test-slave-2kcd krb5kdc[8733](info): set up 2 sockets
Aug 08 04:49:42 dist-test-slave-2kcd krb5kdc[8733](info): commencing operation
krb5kdc: starting...
W20260808 04:49:43.965683 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:44 dist-test-slave-2kcd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:44.090977 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:43687
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:44.208901 8755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:44.209220 8755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:44.209303 8755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:44.213792 8755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:44.213886 8755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:44.213930 8755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:44.213989 8755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:44.214033 8755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:44.219810 8755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:43687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:44.221032 8755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:44.222107 8755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:44.231338 8769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:44.231316 8765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:44.231371 8766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:44.231594 8755 server_base.cc:1034] running on GCE node
I20260808 04:49:44.232283 8755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:44.233392 8755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:44.234553 8755 hybrid_clock.cc:648] HybridClock initialized: now 1786164584234525 us; error 39 us; skew 500 ppm
Aug 08 04:49:44 dist-test-slave-2kcd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164584, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:44.237344 8755 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:44.238183 8755 webserver.cc:460] Webserver started at http://127.4.217.62:41333/ using document root <none> and password file <none>
I20260808 04:49:44.238740 8755 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:44.238811 8755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:44.239059 8755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:44.241175 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "e7d1f07adf8b4cd69965a88d1242b7d9"
format_stamp: "Formatted at 2026-08-08 04:49:44 on dist-test-slave-2kcd"
server_key: "b701574edfc24d944abc6d4dcd2ce19e"
server_key_iv: "322ea959c8be173d56d1b5d9e67521bd"
server_key_version: "encryptionkey@0"
I20260808 04:49:44.241789 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "e7d1f07adf8b4cd69965a88d1242b7d9"
format_stamp: "Formatted at 2026-08-08 04:49:44 on dist-test-slave-2kcd"
server_key: "b701574edfc24d944abc6d4dcd2ce19e"
server_key_iv: "322ea959c8be173d56d1b5d9e67521bd"
server_key_version: "encryptionkey@0"
I20260808 04:49:44.245442 8755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:49:44.247859 8780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:44.249147 8755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:44.249289 8755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "e7d1f07adf8b4cd69965a88d1242b7d9"
format_stamp: "Formatted at 2026-08-08 04:49:44 on dist-test-slave-2kcd"
server_key: "b701574edfc24d944abc6d4dcd2ce19e"
server_key_iv: "322ea959c8be173d56d1b5d9e67521bd"
server_key_version: "encryptionkey@0"
I20260808 04:49:44.249401 8755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:44.256018 8755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:44.259949 8755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:44.260108 8755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:44.267526 8755 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:43687
I20260808 04:49:44.267599 8864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:43687 every 8 connection(s)
I20260808 04:49:44.268513 8755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:44.271384 8871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:44.277263 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 8755
I20260808 04:49:44.277436 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:44.277794 4964 external_mini_cluster.cc:1428] Setting key 9d2b7d64f5e867be60964767e706cbb4
I20260808 04:49:44.277911 8871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9: Bootstrap starting.
I20260808 04:49:44.280833 8871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:44.281754 8871 log.cc:824] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:44.283756 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:44.279298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37929 (local address 127.4.217.62:43687)
0808 04:49:44.279796 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:44.279815 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:44.279858 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:49:44.280702 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:44.280728 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:44.281164 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:44.281534 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:44.282386 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:44.282391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:44.282402 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:44.282411 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:44.282462 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:44.282481 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:44.282596 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:44.282723 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 04:49:44.282877 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":115,"threads_started":1}
I20260808 04:49:44.284071 8871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9: No bootstrap required, opened a new log
I20260808 04:49:44.286788 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:44.288329 8871 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d1f07adf8b4cd69965a88d1242b7d9" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43687 } }
I20260808 04:49:44.288599 8871 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:49:44.288668 8871 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d1f07adf8b4cd69965a88d1242b7d9, State: Initialized, Role: FOLLOWER
I20260808 04:49:44.289222 8871 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d1f07adf8b4cd69965a88d1242b7d9" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43687 } }
I20260808 04:49:44.289392 8871 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:49:44.289474 8871 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:49:44.289589 8871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:44.290594 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:44.288917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41705 (local address 127.4.217.62:43687)
0808 04:49:44.289055 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:44.289059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:44.289224 (+ 165us) server_negotiation.cc:407] Connection header received
0808 04:49:44.289309 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:44.289312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:44.289357 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:44.289442 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:44.290231 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:44.290233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:44.290236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:49:44.290247 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:44.290273 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:49:44.290283 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:44.290353 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:44.290470 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 04:49:44.290502 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:49:44.290925 8871 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d1f07adf8b4cd69965a88d1242b7d9" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43687 } }
I20260808 04:49:44.291318 8871 leader_election.cc:302] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7d1f07adf8b4cd69965a88d1242b7d9; no voters:
I20260808 04:49:44.291667 8871 leader_election.cc:288] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:49:44.291775 8888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:44.292048 8888 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [term 1 LEADER]: Becoming Leader. State: Replica: e7d1f07adf8b4cd69965a88d1242b7d9, State: Running, Role: LEADER
I20260808 04:49:44.292354 8888 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d1f07adf8b4cd69965a88d1242b7d9" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43687 } }
I20260808 04:49:44.293047 8871 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:44.294170 8890 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7d1f07adf8b4cd69965a88d1242b7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d1f07adf8b4cd69965a88d1242b7d9" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43687 } } }
I20260808 04:49:44.294323 8890 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:44.294170 8891 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d1f07adf8b4cd69965a88d1242b7d9. Latest consensus state: current_term: 1 leader_uuid: "e7d1f07adf8b4cd69965a88d1242b7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d1f07adf8b4cd69965a88d1242b7d9" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 43687 } } }
I20260808 04:49:44.294615 8891 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:44.294836 8901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:44.298125 8901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:44.304319 8901 catalog_manager.cc:1347] Generated new cluster ID: 1914d2de241e4713aa2bfef098f8f1a1
I20260808 04:49:44.304425 8901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:44.342111 8901 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:44.342475 8901 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:44.348213 8901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7d1f07adf8b4cd69965a88d1242b7d9: Generated new TSK 0
I20260808 04:49:44.348914 8901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:49:44.352751 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 8755
2026-08-08T04:49:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1675 ms)
[----------] 18 tests from AuthTokenIssuingTest (29696 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:44 dist-test-slave-2kcd krb5kdc[8929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:44 dist-test-slave-2kcd krb5kdc[8929](info): set up 2 sockets
Aug 08 04:49:44 dist-test-slave-2kcd krb5kdc[8929](info): commencing operation
krb5kdc: starting...
W20260808 04:49:45.630450 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:45 dist-test-slave-2kcd krb5kdc[8929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:45.756294 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35039
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35039,127.4.217.61:41011,127.4.217.60:41971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:45.869112 8947 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:49:45.869364 8947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:45.869386 8947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:45.869469 8947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:45.873390 8947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:45.873437 8947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:45.873456 8947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:45.873521 8947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:45.873560 8947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:45.879282 8947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35039,127.4.217.61:41011,127.4.217.60:41971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:35039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:45.880250 8947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:45.881249 8947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:45.889633 8958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:45.889595 8955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:45.889605 8960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:45.889694 8947 server_base.cc:1034] running on GCE node
I20260808 04:49:45.890390 8947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:45.891429 8947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:45.892634 8947 hybrid_clock.cc:648] HybridClock initialized: now 1786164585892619 us; error 34 us; skew 500 ppm
Aug 08 04:49:45 dist-test-slave-2kcd krb5kdc[8929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164585, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:45.895262 8947 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:45.896101 8947 webserver.cc:460] Webserver started at http://127.4.217.62:42457/ using document root <none> and password file <none>
I20260808 04:49:45.896662 8947 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:45.896770 8947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:45.897013 8947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:45.898985 8947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "bca1895c41304e68b7d079ed1c990062"
format_stamp: "Formatted at 2026-08-08 04:49:45 on dist-test-slave-2kcd"
server_key: "4f9b2d4a791ea79ce48d8c80a3743399"
server_key_iv: "9f2af9e5e0ab338b5c72c9379ea7327a"
server_key_version: "encryptionkey@0"
I20260808 04:49:45.899554 8947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "bca1895c41304e68b7d079ed1c990062"
format_stamp: "Formatted at 2026-08-08 04:49:45 on dist-test-slave-2kcd"
server_key: "4f9b2d4a791ea79ce48d8c80a3743399"
server_key_iv: "9f2af9e5e0ab338b5c72c9379ea7327a"
server_key_version: "encryptionkey@0"
I20260808 04:49:45.903216 8947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:49:45.905642 8975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:45.906793 8947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:49:45.906971 8947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "bca1895c41304e68b7d079ed1c990062"
format_stamp: "Formatted at 2026-08-08 04:49:45 on dist-test-slave-2kcd"
server_key: "4f9b2d4a791ea79ce48d8c80a3743399"
server_key_iv: "9f2af9e5e0ab338b5c72c9379ea7327a"
server_key_version: "encryptionkey@0"
I20260808 04:49:45.907088 8947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:45.927057 8947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:45.931433 8947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:45.931639 8947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:45.938624 8947 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:35039
I20260808 04:49:45.938897 9057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:35039 every 8 connection(s)
I20260808 04:49:45.939604 8947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:45.942417 9059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:45.942836 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 8947
I20260808 04:49:45.943017 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:45.943308 4964 external_mini_cluster.cc:1428] Setting key 65b1076053348db6cea7a6aa895e19b3
I20260808 04:49:45.946905 9059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.61@KRBTEST.COM" created.
Aug 08 04:49:45 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164585, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:45.962510 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:45.950417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.4.217.62:35039)
0808 04:49:45.950737 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:45.950749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:45.950793 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:49:45.951136 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:45.951140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:45.951312 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:45.951523 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:45.952327 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:45.953430 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:45.954479 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:45.954848 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:45.955086 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:45.957667 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:45.957673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:45.957683 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:45.957703 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:45.959329 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:45.959838 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:45.959843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:45.959845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:45.959954 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:45.960301 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:45.960304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:45.960306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:45.960662 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:45.960839 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:45.961231 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 04:49:45.961480 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":98,"threads_started":1}
I20260808 04:49:45.962510 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:45.949400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35039 (local address 127.0.0.1:50118)
0808 04:49:45.949913 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:45.949954 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:45.950803 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:45.951499 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:45.951505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:45.951925 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:45.952173 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:45.952186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:45.953575 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:45.953578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:45.954354 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:45.954365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:45.955071 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:45.955075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:45.955159 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:45.955828 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:45.955844 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:45.957488 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:45.959464 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:45.959475 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:45.959484 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:45.959718 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:45.960072 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:45.960076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:45.960078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:45.960194 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:45.960828 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:45.960835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:45.960977 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 04:49:45.961156 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 04:49:45.961567 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":110,"thread_start_us":145,"threads_started":1}
I20260808 04:49:45.965718 9059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:45.966498 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:45.966207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:41011 (local address 127.0.0.1:59770)
0808 04:49:45.966343 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:45.966430 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260808 04:49:45.966805 8980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
I20260808 04:49:45.967736 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:45.967525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:41011 (local address 127.0.0.1:59778)
0808 04:49:45.967634 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:45.967681 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:49:45.967990 9059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.61:41011: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab.
Entry for principal kudu/127.4.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab.
I20260808 04:49:45.972034 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:41011
--webserver_interface=127.4.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab
--principal=kudu/127.4.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35039,127.4.217.61:41011,127.4.217.60:41971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:49:46.017267 9059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } attempt: 1
I20260808 04:49:46.018312 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.017945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:41011 (local address 127.0.0.1:59788)
0808 04:49:46.018135 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.018212 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 04:49:46.019101 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.018849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:41011 (local address 127.0.0.1:59794)
0808 04:49:46.018972 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.019026 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 04:49:46.019260 9059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.61:41011: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
W20260808 04:49:46.085390 9073 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:49:46.085647 9073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:46.085670 9073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:46.085748 9073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 04:49:46.087448 9059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } attempt: 2
I20260808 04:49:46.088337 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.087970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:41011 (local address 127.0.0.1:59806)
0808 04:49:46.088167 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.088242 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 04:49:46.089181 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.088934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:41011 (local address 127.0.0.1:59820)
0808 04:49:46.089059 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.089115 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:49:46.089357 9059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.61:41011: Network error: Client connection negotiation failed: client connection to 127.4.217.61:41011: connect: Connection refused (error 111)
W20260808 04:49:46.090931 9073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:46.090982 9073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:46.091030 9073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:46.091068 9073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:46.091130 9073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:46.096614 9073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35039,127.4.217.61:41011,127.4.217.60:41971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab
--principal=kudu/127.4.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:41011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:46.097580 9073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:46.098575 9073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:46.106940 9089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:46.106881 9092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:46.106880 9090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:46.106938 9073 server_base.cc:1034] running on GCE node
I20260808 04:49:46.107699 9073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:46.108670 9073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:46.109844 9073 hybrid_clock.cc:648] HybridClock initialized: now 1786164586109811 us; error 40 us; skew 500 ppm
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:46.117460 9073 init.cc:377] Logged in from keytab as kudu/127.4.217.61@KRBTEST.COM (short username kudu)
I20260808 04:49:46.118419 9073 webserver.cc:460] Webserver started at http://127.4.217.61:45111/ using document root <none> and password file <none>
I20260808 04:49:46.118917 9073 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:46.118974 9073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:46.119140 9073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:46.121248 9073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/data/instance:
uuid: "6ee67c6269794fe6bb53af53e84cea76"
format_stamp: "Formatted at 2026-08-08 04:49:46 on dist-test-slave-2kcd"
server_key: "c6e2f25ce07589154c7ff4186e43f117"
server_key_iv: "ca5c93c564137b395373c785a4b3652e"
server_key_version: "encryptionkey@0"
I20260808 04:49:46.121770 9073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/wal/instance:
uuid: "6ee67c6269794fe6bb53af53e84cea76"
format_stamp: "Formatted at 2026-08-08 04:49:46 on dist-test-slave-2kcd"
server_key: "c6e2f25ce07589154c7ff4186e43f117"
server_key_iv: "ca5c93c564137b395373c785a4b3652e"
server_key_version: "encryptionkey@0"
I20260808 04:49:46.125566 9073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:49:46.127987 9103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:46.129280 9073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:46.129429 9073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/wal
uuid: "6ee67c6269794fe6bb53af53e84cea76"
format_stamp: "Formatted at 2026-08-08 04:49:46 on dist-test-slave-2kcd"
server_key: "c6e2f25ce07589154c7ff4186e43f117"
server_key_iv: "ca5c93c564137b395373c785a4b3652e"
server_key_version: "encryptionkey@0"
I20260808 04:49:46.129542 9073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:46.145993 9073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:46.148904 9073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:46.149107 9073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:46.156337 9073 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:41011
I20260808 04:49:46.156641 9187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.61:41011 every 8 connection(s)
I20260808 04:49:46.157361 9073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
I20260808 04:49:46.158475 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9073
I20260808 04:49:46.158607 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-1/wal/instance
I20260808 04:49:46.158917 4964 external_mini_cluster.cc:1428] Setting key ecc8d876ca5fa33f6655de324469db3d
I20260808 04:49:46.160771 9188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:46.164724 9188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.217.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.60@KRBTEST.COM" created.
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:46.180560 9059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } attempt: 3
I20260808 04:49:46.180930 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.166710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.4.217.62:35039)
0808 04:49:46.166900 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.166907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.167657 (+ 750us) server_negotiation.cc:407] Connection header received
0808 04:49:46.168648 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.168651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.168743 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.168876 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.170047 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.170754 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.172530 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.172969 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.173097 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.176026 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.176029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.176031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.176055 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.178342 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.179019 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.179022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.179023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.179108 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.179515 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.179518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.179519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.179708 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.179786 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.180389 (+ 603us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.180502 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:49:46.182066 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.167020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35039 (local address 127.0.0.1:50120)
0808 04:49:46.167505 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.167557 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:46.168425 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:46.169034 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:46.169041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:46.169358 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:46.169904 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.169921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.170896 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.170899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.172371 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.172387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.173129 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.173132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.173270 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.173943 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:46.173959 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:46.175831 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:46.178488 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.178495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.178510 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.178901 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.179229 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.179233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.179235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.179417 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.179851 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:46.179860 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:46.180144 (+ 284us) client_negotiation.cc:769] Sending connection context
0808 04:49:46.180313 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 04:49:46.180673 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":152,"threads_started":1}
I20260808 04:49:46.183822 9188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164585, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab.
Entry for principal kudu/127.4.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab.
I20260808 04:49:46.188984 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:41971
--webserver_interface=127.4.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab
--principal=kudu/127.4.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:35039,127.4.217.61:41011,127.4.217.60:41971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
I20260808 04:49:46.192301 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.181075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:41011 (local address 127.0.0.1:59836)
0808 04:49:46.181209 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.181222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:46.181335 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:46.182411 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:46.182419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:46.182608 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:46.182740 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.182747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.184052 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.184056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.184931 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.184941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.185736 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.185740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.185856 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.186392 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:46.186407 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:46.187971 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:46.190498 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.190505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.190510 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.190785 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.191163 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.191167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.191169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.191284 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.191741 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:46.191744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:46.191843 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 04:49:46.191969 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 04:49:46.192110 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 04:49:46.192487 9194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.181453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.4.217.61:41011)
0808 04:49:46.181805 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.181818 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.181870 (+ 52us) server_negotiation.cc:407] Connection header received
0808 04:49:46.181926 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.181937 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.182167 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.182426 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.182905 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.183843 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.185066 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.185615 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.185735 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.188127 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.188137 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.188150 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.188178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.190354 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.190933 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.190941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.190942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.191048 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.191387 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.191389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.191390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.191614 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.191777 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.192090 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.192272 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":256,"thread_start_us":145,"threads_started":1}
I20260808 04:49:46.194273 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.184360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:41011 (local address 127.0.0.1:59844)
0808 04:49:46.184686 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.184702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:46.184824 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:46.185072 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:46.185074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:46.185246 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:46.185364 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.185371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.186309 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.186316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.187088 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.187098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.187819 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.187823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.187943 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.188442 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:46.188462 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:46.190057 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:46.192611 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.192615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.192618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.192890 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.193203 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.193209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.193212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.193314 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.193763 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:46.193766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:46.193855 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 04:49:46.193959 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 04:49:46.194107 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":19}
I20260808 04:49:46.194625 9059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:46.194329 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.184483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.4.217.61:41011)
0808 04:49:46.184832 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.184840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.184854 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:49:46.184912 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.184915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.184957 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.185043 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.185480 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.186208 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.187215 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.187697 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.187821 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.190206 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.190210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.190212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.190242 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.192492 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.192997 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.193001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.193002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.193092 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.193451 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.193456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.193461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.193656 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.193773 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.194018 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.194164 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":248,"thread_start_us":136,"threads_started":1}
I20260808 04:49:46.195653 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.195135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40248)
0808 04:49:46.195300 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.195567 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 04:49:46.195742 9188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:46.196717 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.196432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40276)
0808 04:49:46.196568 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.196653 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 04:49:46.196887 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.196379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40264)
0808 04:49:46.196690 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.196750 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":122,"threads_started":1}
W20260808 04:49:46.196916 9106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
W20260808 04:49:46.197034 9059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:41971: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
I20260808 04:49:46.197889 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.197676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40280)
0808 04:49:46.197786 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.197831 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:49:46.198053 9188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:41971: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
I20260808 04:49:46.228209 9059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } attempt: 1
I20260808 04:49:46.229056 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.228710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40292)
0808 04:49:46.228899 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.228964 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 04:49:46.229781 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.229559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40304)
0808 04:49:46.229667 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.229720 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:49:46.229919 9059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:41971: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
I20260808 04:49:46.247272 9188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } attempt: 1
I20260808 04:49:46.248097 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.247772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40308)
0808 04:49:46.247942 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.248011 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 04:49:46.248907 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.248648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40320)
0808 04:49:46.248763 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.248837 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:49:46.249044 9188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:41971: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
I20260808 04:49:46.305101 9059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } attempt: 2
I20260808 04:49:46.306018 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.305655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40326)
0808 04:49:46.305845 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.305919 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 04:49:46.306815 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.306586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40332)
0808 04:49:46.306703 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.306754 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:49:46.306958 9059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:41971: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
W20260808 04:49:46.313130 9197 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:49:46.313371 9197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:46.313393 9197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:46.313436 9197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 04:49:46.317195 9188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } attempt: 2
W20260808 04:49:46.317471 9197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:46.317518 9197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:46.317538 9197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:46.317576 9197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:46.317611 9197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:46.318020 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.317723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40348)
0808 04:49:46.317876 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.317939 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 04:49:46.318768 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.318534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40350)
0808 04:49:46.318650 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.318702 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:49:46.318903 9188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:41971: Network error: Client connection negotiation failed: client connection to 127.4.217.60:41971: connect: Connection refused (error 111)
I20260808 04:49:46.323261 9197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:35039,127.4.217.61:41011,127.4.217.60:41971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab
--principal=kudu/127.4.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:41971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:46.324314 9197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:46.325462 9197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:46.334252 9207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:46.334252 9211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:46.334255 9208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:46.334613 9197 server_base.cc:1034] running on GCE node
I20260808 04:49:46.335259 9197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:46.336401 9197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:46.337606 9197 hybrid_clock.cc:648] HybridClock initialized: now 1786164586337590 us; error 28 us; skew 500 ppm
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:46.340492 9197 init.cc:377] Logged in from keytab as kudu/127.4.217.60@KRBTEST.COM (short username kudu)
I20260808 04:49:46.341461 9197 webserver.cc:460] Webserver started at http://127.4.217.60:38913/ using document root <none> and password file <none>
I20260808 04:49:46.342047 9197 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:46.342145 9197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:46.342386 9197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:46.344424 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/data/instance:
uuid: "0efd0e0001c34da38b02ce2b57998db9"
format_stamp: "Formatted at 2026-08-08 04:49:46 on dist-test-slave-2kcd"
server_key: "4f4099df4066ef50735b0bf211ae4482"
server_key_iv: "22df101d6a74b483fb01edf8255a1987"
server_key_version: "encryptionkey@0"
I20260808 04:49:46.345041 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/wal/instance:
uuid: "0efd0e0001c34da38b02ce2b57998db9"
format_stamp: "Formatted at 2026-08-08 04:49:46 on dist-test-slave-2kcd"
server_key: "4f4099df4066ef50735b0bf211ae4482"
server_key_iv: "22df101d6a74b483fb01edf8255a1987"
server_key_version: "encryptionkey@0"
I20260808 04:49:46.348875 9197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:49:46.351300 9221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:46.352545 9197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:49:46.352689 9197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/wal
uuid: "0efd0e0001c34da38b02ce2b57998db9"
format_stamp: "Formatted at 2026-08-08 04:49:46 on dist-test-slave-2kcd"
server_key: "4f4099df4066ef50735b0bf211ae4482"
server_key_iv: "22df101d6a74b483fb01edf8255a1987"
server_key_version: "encryptionkey@0"
I20260808 04:49:46.352849 9197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:46.363838 9197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:46.366873 9197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:46.367080 9197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:46.374600 9197 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.60:41971
I20260808 04:49:46.374917 9314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.60:41971 every 8 connection(s)
I20260808 04:49:46.375569 9197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
I20260808 04:49:46.378391 9315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:46.378993 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9197
I20260808 04:49:46.379107 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164296366965-4964-0/minicluster-data/master-2/wal/instance
I20260808 04:49:46.379410 4964 external_mini_cluster.cc:1428] Setting key 656ab3f56a4cc57a597121d83b846ea8
I20260808 04:49:46.383055 9315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:46.395047 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.385089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.4.217.62:35039)
0808 04:49:46.385246 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.385252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.385540 (+ 288us) server_negotiation.cc:407] Connection header received
0808 04:49:46.385688 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.385690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.385734 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.385803 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.386617 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.387332 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.388696 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.389014 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.389140 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.391577 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.391580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.391582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.391604 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.393180 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.393716 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.393719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.393721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.393813 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.394162 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.394165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.394167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.394388 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.394502 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.394796 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.394908 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:49:46.395071 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.385011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:35039 (local address 127.0.0.1:50136)
0808 04:49:46.385393 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.385444 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:46.385565 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:46.385889 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:46.385894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:46.386328 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:46.386492 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.386500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.387476 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.387481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.388552 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.388583 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.389208 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.389212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.389288 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.389796 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:46.389812 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:46.391412 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:46.393285 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.393293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.393297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.393603 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.393930 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.393936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.393937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.394049 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.394506 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:46.394510 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:46.394603 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 04:49:46.394711 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 04:49:46.394830 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
I20260808 04:49:46.395078 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.380841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.4.217.60:41971)
0808 04:49:46.381286 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.381306 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.381350 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:49:46.382198 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.382222 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.382674 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.383016 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.383768 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.385148 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.386389 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.387036 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.387195 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.390149 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.390160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.390170 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.390193 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.391859 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.392313 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.392322 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.392331 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.392478 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.392774 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.392800 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.392802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.393335 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.393491 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.393841 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.394095 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260808 04:49:46.396574 9315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:46.405474 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.397079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:41011 (local address 127.0.0.1:59848)
0808 04:49:46.397296 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.397307 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:46.397414 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:46.397942 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:46.397944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:46.398079 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:46.398167 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.398171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.398927 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.398932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.399423 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.399432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.399916 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.399920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.400030 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.400467 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:46.400478 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:46.401838 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:46.403927 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.403929 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.403931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.404125 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.404435 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.404438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.404440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.404548 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.405011 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:46.405015 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:46.405116 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 04:49:46.405259 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 04:49:46.405368 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260808 04:49:46.405619 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.397190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.4.217.61:41011)
0808 04:49:46.397361 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.397366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.397392 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:49:46.397524 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.397528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.397600 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.397708 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.398293 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.398812 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.399559 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.399804 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.399914 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.402002 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.402005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.402007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.402043 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.403805 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.404227 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.404233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.404234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.404323 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.404646 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.404648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.404649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.404892 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.405007 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.405323 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.405447 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:49:46.406104 9059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } attempt: 3
I20260808 04:49:46.406395 9315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:46.406498 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.397659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.4.217.62:35039)
0808 04:49:46.397794 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.397797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.397809 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:49:46.397918 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.397920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.397953 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.398057 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.398614 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.399063 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.399808 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.400023 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.400136 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.402657 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.402663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.402668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.402698 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.404296 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.404776 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.404798 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.404800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.404880 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.405246 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.405252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.405256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.405456 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.405887 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.406104 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.406348 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:49:46.410051 9188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } attempt: 3
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164585, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
Aug 08 04:49:46 dist-test-slave-2kcd krb5kdc[8929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
I20260808 04:49:46.417608 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.406487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40374)
0808 04:49:46.406780 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.406794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:46.406862 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:46.407502 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:46.407506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:46.407686 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:46.407787 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.407794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.409195 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.409199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.409980 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.409990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.411519 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.411533 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.411632 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.412254 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:46.412267 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:46.413715 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:46.415870 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.415874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.415877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.416119 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.416467 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.416472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.416475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.416586 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.417098 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:46.417101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:46.417198 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 04:49:46.417323 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 04:49:46.417447 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260808 04:49:46.417635 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.406662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.4.217.60:41971)
0808 04:49:46.406993 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.406997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.407009 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:49:46.407054 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.407059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.407107 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.407485 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.407918 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.409092 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.410443 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.411371 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.411509 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.413872 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.413876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.413878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.413904 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.415740 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.416252 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.416257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.416262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.416351 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.416720 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.416723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.416724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.416963 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.417104 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.417378 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.417480 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":817,"server-negotiator.queue_time_us":268,"spinlock_wait_cycles":11904}
I20260808 04:49:46.419714 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.407830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.4.217.61:41011)
0808 04:49:46.407931 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.407937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.407952 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:49:46.407990 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.407994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.408040 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.408136 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.408716 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.409474 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.410519 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.410853 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.410939 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.415347 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.415352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.415355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.415382 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.417157 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.418230 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.418234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.418236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.418331 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.418699 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.418706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.418710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.418919 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.419139 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.419291 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.419428 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 04:49:46.420107 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.406820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40386)
0808 04:49:46.407220 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.407232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:46.407292 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:46.407639 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:46.407643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:46.407800 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:46.407888 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.407895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.409001 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.409004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.409594 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.409604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.410557 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.410560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.410646 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.411475 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:46.411492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:46.412942 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:46.418183 (+ 5241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.418186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.418188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.418670 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.418954 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.418960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.418965 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.419059 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.419574 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:46.419579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:46.419663 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 04:49:46.419768 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:49:46.419902 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344}
I20260808 04:49:46.420222 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.406985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.4.217.60:41971)
0808 04:49:46.407300 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.407307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.407320 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:49:46.407347 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.407349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.407417 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.407514 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.407990 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.408893 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.409715 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.410340 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.410436 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.413082 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.413086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.413089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.413111 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.417921 (+ 4810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.418755 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.418759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.418760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.418849 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.419162 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.419165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.419179 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.419374 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.419511 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.419856 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.419982 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":261,"spinlock_wait_cycles":896,"thread_start_us":123,"threads_started":1}
I20260808 04:49:46.421510 9059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062: Bootstrap starting.
I20260808 04:49:46.422742 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.410420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:41971 (local address 127.0.0.1:40400)
0808 04:49:46.410580 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:46.410593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:46.410659 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:46.411019 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:46.411032 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:46.411203 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:46.411322 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.411333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.412511 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.412515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.413472 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:46.413484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.414077 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.414081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:46.414176 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.414656 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:46.414674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:46.416204 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:46.421187 (+ 4983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.421190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.421192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.421433 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.421787 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:46.421790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:46.421792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:46.421879 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:46.422319 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:46.422322 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:46.422407 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 04:49:46.422484 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:49:46.422605 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 04:49:46.423270 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:46.423293 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.410487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.4.217.60:41971)
0808 04:49:46.410783 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.410791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.410804 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:49:46.410835 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.410838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.410874 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.410994 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.411441 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.412385 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.413588 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.413936 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.414429 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.416351 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.416356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.416360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.416389 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.418584 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.421566 (+ 2982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.421569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.421570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.421676 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.421996 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.422000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.422001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.422199 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.422911 (+ 712us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.422991 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.423131 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
I20260808 04:49:46.423988 9315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9: Bootstrap starting.
I20260808 04:49:46.424343 9059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:46.425303 9059 log.cc:824] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:46.424719 9015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bca1895c41304e68b7d079ed1c990062"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152
I20260808 04:49:46.426520 9315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:46.426954 9188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76: Bootstrap starting.
I20260808 04:49:46.427449 9315 log.cc:824] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:46.427959 9059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062: No bootstrap required, opened a new log
I20260808 04:49:46.429165 9188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:46.429733 9315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9: No bootstrap required, opened a new log
I20260808 04:49:46.430006 9188 log.cc:824] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:46.431026 9059 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } }
I20260808 04:49:46.431253 9059 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca1895c41304e68b7d079ed1c990062, State: Initialized, Role: FOLLOWER
I20260808 04:49:46.431576 9059 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } }
I20260808 04:49:46.432072 9331 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.432312 9331 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:46.432955 9059 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:46.432986 9015 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
W20260808 04:49:46.433071 8984 tablet.cc:2369] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:49:46.433132 9015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:46.433887 9315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } }
I20260808 04:49:46.434168 9315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efd0e0001c34da38b02ce2b57998db9, State: Initialized, Role: FOLLOWER
I20260808 04:49:46.434408 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } }
I20260808 04:49:46.434494 9015 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } }
I20260808 04:49:46.434778 9340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.434890 9340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:46.435590 9315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:46.436120 9015 leader_election.cc:288] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [CANDIDATE]: Term 1 election: Requested vote from peers 6ee67c6269794fe6bb53af53e84cea76 (127.4.217.61:41011), 0efd0e0001c34da38b02ce2b57998db9 (127.4.217.60:41971)
I20260808 04:49:46.436970 9268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca1895c41304e68b7d079ed1c990062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0efd0e0001c34da38b02ce2b57998db9"
I20260808 04:49:46.437182 9268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:46.438503 9268 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bca1895c41304e68b7d079ed1c990062 in term 1.
I20260808 04:49:46.438930 8978 leader_election.cc:302] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0efd0e0001c34da38b02ce2b57998db9, bca1895c41304e68b7d079ed1c990062; no voters:
I20260808 04:49:46.439189 9331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:46.439320 9188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76: No bootstrap required, opened a new log
I20260808 04:49:46.439405 9331 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [term 1 LEADER]: Becoming Leader. State: Replica: bca1895c41304e68b7d079ed1c990062, State: Running, Role: LEADER
I20260808 04:49:46.439718 9331 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } }
I20260808 04:49:46.441416 9331 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca1895c41304e68b7d079ed1c990062. Latest consensus state: current_term: 1 leader_uuid: "bca1895c41304e68b7d079ed1c990062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.441524 9331 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:46.440160 8947 master_runner.cc:186] Error getting master registration for 127.4.217.61:41011: OK, instance_id { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" instance_seqno: 1786164586154479 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:49:46.443202 9188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } }
I20260808 04:49:46.443388 8947 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:49:46.443390 9188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee67c6269794fe6bb53af53e84cea76, State: Initialized, Role: FOLLOWER
I20260808 04:49:46.443713 9188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } }
I20260808 04:49:46.444221 9388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.444850 9388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:46.445070 9188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:46.445142 9145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca1895c41304e68b7d079ed1c990062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ee67c6269794fe6bb53af53e84cea76"
I20260808 04:49:46.445344 9145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:46.446619 9145 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bca1895c41304e68b7d079ed1c990062 in term 1.
W20260808 04:49:46.446714 9356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:49:46.446782 9356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:49:46.447240 9387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:46.449556 9387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260808 04:49:46.450798 9414 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:49:46.450872 9414 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260808 04:49:46.451193 9415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:49:46.453019 9415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:49:46.463472 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.448128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.4.217.61:41011)
0808 04:49:46.448235 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.448239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.452274 (+ 4035us) server_negotiation.cc:407] Connection header received
0808 04:49:46.452442 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.452446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.452489 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.452542 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.455634 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.456183 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.457126 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.457699 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.457796 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.459688 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.459692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.459695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.459723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.461520 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.462043 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.462047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.462049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.462135 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.462464 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.462468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.462469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.462679 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.462795 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.463212 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.463327 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 04:49:46.463495 9268 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [term 1 FOLLOWER]: Refusing update from remote peer bca1895c41304e68b7d079ed1c990062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:46.463819 9145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [term 1 FOLLOWER]: Refusing update from remote peer bca1895c41304e68b7d079ed1c990062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:46.464200 9331 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:46.464639 9331 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:46.468600 9388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca1895c41304e68b7d079ed1c990062. Latest consensus state: current_term: 1 leader_uuid: "bca1895c41304e68b7d079ed1c990062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.468747 9388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:46.469200 9340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca1895c41304e68b7d079ed1c990062. Latest consensus state: current_term: 1 leader_uuid: "bca1895c41304e68b7d079ed1c990062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.469312 9340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:46.473635 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.438408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.4.217.60:41971)
0808 04:49:46.438543 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.438548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.452189 (+ 13641us) server_negotiation.cc:407] Connection header received
0808 04:49:46.452235 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.452238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.452287 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.452370 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.456105 (+ 3735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.456975 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.466609 (+ 9634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.466968 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.467051 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.469990 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.469994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.469997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.470023 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.471785 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.472268 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.472272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.472273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.472357 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.472667 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.472670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.472672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.472911 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.473024 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.473353 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.473469 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:49:46.474792 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:46.447652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.4.217.62:35039)
0808 04:49:46.447774 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.447780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.451950 (+ 4170us) server_negotiation.cc:407] Connection header received
0808 04:49:46.452077 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.452080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.452132 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.452204 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.453948 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.454718 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.465320 (+ 10602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:46.465647 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:46.465727 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:46.470907 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:46.470911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:46.470915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:46.470941 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:46.472678 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.473197 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.473201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.473203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.473290 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:46.473598 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:46.473602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:46.473603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:46.473808 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:46.473914 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:46.474473 (+ 559us) server_negotiation.cc:299] Negotiation successful
0808 04:49:46.474643 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:49:46.479673 9384 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca1895c41304e68b7d079ed1c990062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.479957 9384 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:46.480105 9424 mvcc.cc:205] Tried to move back new op lower bound from 7316130146146291712 to 7316130146060341248. Current Snapshot: MvccSnapshot[applied={T|T < 7316130146146291712}]
I20260808 04:49:46.480608 9331 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca1895c41304e68b7d079ed1c990062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.480645 9340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca1895c41304e68b7d079ed1c990062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.480718 9331 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062 [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:46.480731 9340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:46.481019 9428 mvcc.cc:205] Tried to move back new op lower bound from 7316130146146291712 to 7316130146060341248. Current Snapshot: MvccSnapshot[applied={T|T < 7316130146146291712}]
I20260808 04:49:46.481374 9388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca1895c41304e68b7d079ed1c990062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca1895c41304e68b7d079ed1c990062" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 35039 } } peers { permanent_uuid: "6ee67c6269794fe6bb53af53e84cea76" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 41011 } } peers { permanent_uuid: "0efd0e0001c34da38b02ce2b57998db9" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 41971 } } }
I20260808 04:49:46.481491 9388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:46.481886 9427 mvcc.cc:205] Tried to move back new op lower bound from 7316130146146291712 to 7316130146060341248. Current Snapshot: MvccSnapshot[applied={T|T < 7316130146146291712}]
I20260808 04:49:46.487854 9387 catalog_manager.cc:1347] Generated new cluster ID: de08b65cc5dc446da0082ab99491de34
I20260808 04:49:46.487910 9387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:46.512835 9387 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:46.514153 9387 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:46.531164 9387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca1895c41304e68b7d079ed1c990062: Generated new TSK 0
I20260808 04:49:46.532140 9387 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:49:46.573565 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.570042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.4.217.61:41011)
0808 04:49:46.570176 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.570181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.572378 (+ 2197us) server_negotiation.cc:407] Connection header received
0808 04:49:46.572598 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.572602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.572656 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.572713 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.573412 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59876: BlockingRecv error: recv got EOF from 127.0.0.1:59876 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 04:49:46.575096 9324 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.570836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.4.217.60:41971)
0808 04:49:46.570949 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.570953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.574209 (+ 3256us) server_negotiation.cc:407] Connection header received
0808 04:49:46.574372 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.574376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.574424 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.574471 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.574971 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40416: BlockingRecv error: recv got EOF from 127.0.0.1:40416 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 04:49:46.576673 9066 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.571708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.4.217.62:35039)
0808 04:49:46.571828 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.571832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.575720 (+ 3888us) server_negotiation.cc:407] Connection header received
0808 04:49:46.575944 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.575947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.576012 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.576064 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.576559 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50168: BlockingRecv error: recv got EOF from 127.0.0.1:50168 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 04:49:46.581398 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.579809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.4.217.61:41011)
0808 04:49:46.579906 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.579910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.580395 (+ 485us) server_negotiation.cc:407] Connection header received
0808 04:49:46.580626 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.580629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.580702 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.580766 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.581288 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59882: BlockingRecv error: recv got EOF from 127.0.0.1:59882 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 04:49:46.586266 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.584748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.4.217.61:41011)
0808 04:49:46.584868 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.584874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.585386 (+ 512us) server_negotiation.cc:407] Connection header received
0808 04:49:46.585580 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.585583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.585630 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.585679 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.586178 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59892: BlockingRecv error: recv got EOF from 127.0.0.1:59892 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 04:49:46.591778 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.590214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.4.217.61:41011)
0808 04:49:46.590329 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.590335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.590890 (+ 555us) server_negotiation.cc:407] Connection header received
0808 04:49:46.591074 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.591079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.591130 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.591186 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.591695 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59900: BlockingRecv error: recv got EOF from 127.0.0.1:59900 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 04:49:46.599481 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.597925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.4.217.61:41011)
0808 04:49:46.598054 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.598060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.598599 (+ 539us) server_negotiation.cc:407] Connection header received
0808 04:49:46.598784 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.598789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.598836 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.598885 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.599389 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59912: BlockingRecv error: recv got EOF from 127.0.0.1:59912 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 04:49:46.611871 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.609885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.4.217.61:41011)
0808 04:49:46.610002 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.610006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.610719 (+ 713us) server_negotiation.cc:407] Connection header received
0808 04:49:46.610957 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.610961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.611020 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.611079 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.611764 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59922: BlockingRecv error: recv got EOF from 127.0.0.1:59922 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 04:49:46.632248 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.630435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.4.217.61:41011)
0808 04:49:46.630566 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.630570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.631175 (+ 605us) server_negotiation.cc:407] Connection header received
0808 04:49:46.631392 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.631395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.631463 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.631520 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.632133 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59928: BlockingRecv error: recv got EOF from 127.0.0.1:59928 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 04:49:46.668200 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.666825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.4.217.61:41011)
0808 04:49:46.666958 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.666962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.667237 (+ 275us) server_negotiation.cc:407] Connection header received
0808 04:49:46.667426 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.667430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.667490 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.667590 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.668075 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59934: BlockingRecv error: recv got EOF from 127.0.0.1:59934 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 04:49:46.735762 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.734560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.4.217.61:41011)
0808 04:49:46.734728 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.734732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.734812 (+ 80us) server_negotiation.cc:407] Connection header received
0808 04:49:46.734938 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.734940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.734989 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.735127 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.735629 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59936: BlockingRecv error: recv got EOF from 127.0.0.1:59936 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 04:49:46.867385 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:46.866093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.4.217.61:41011)
0808 04:49:46.866268 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:46.866274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:46.866412 (+ 138us) server_negotiation.cc:407] Connection header received
0808 04:49:46.866600 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:46.866608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:46.866657 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:46.866751 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:46.867249 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59938: BlockingRecv error: recv got EOF from 127.0.0.1:59938 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 04:49:47.126793 9195 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:47.125608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.4.217.61:41011)
0808 04:49:47.125770 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:47.125778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:47.125817 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:49:47.126009 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:47.126013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:47.126087 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:47.126181 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:47.126674 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59946: BlockingRecv error: recv got EOF from 127.0.0.1:59946 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:49:47.452147 9414 catalog_manager.cc:1259] Loaded cluster ID: de08b65cc5dc446da0082ab99491de34
I20260808 04:49:47.452236 9414 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76: loading cluster ID for follower catalog manager: success
I20260808 04:49:47.454277 9415 catalog_manager.cc:1259] Loaded cluster ID: de08b65cc5dc446da0082ab99491de34
I20260808 04:49:47.454352 9415 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9: loading cluster ID for follower catalog manager: success
I20260808 04:49:47.454349 9414 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76: acquiring CA information for follower catalog manager: success
I20260808 04:49:47.455744 9414 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ee67c6269794fe6bb53af53e84cea76: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:49:47.456269 9415 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9: acquiring CA information for follower catalog manager: success
I20260808 04:49:47.457319 9415 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0efd0e0001c34da38b02ce2b57998db9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:49:47.645378 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:47.641388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.4.217.61:41011)
0808 04:49:47.641672 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:47.641677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:47.641693 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:49:47.641787 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:47.641790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:47.641925 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:47.642028 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:47.642323 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.642854 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.643737 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.643960 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.644038 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:47.644337 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:47.644876 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:47.644976 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:47.645106 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 04:49:47.645163 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260808 04:49:47.653146 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:47.649290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.4.217.60:41971)
0808 04:49:47.649571 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:47.649574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:47.649589 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:49:47.649678 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:47.649680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:47.649801 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:47.649911 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:47.650131 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.650653 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.651470 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.651696 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.651788 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:47.652094 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:47.652566 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:47.652674 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:47.652802 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 04:49:47.652928 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
I20260808 04:49:47.661651 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:47.657499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.4.217.61:41011)
0808 04:49:47.657704 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:47.657708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:47.657722 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:49:47.657795 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:47.657801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:47.657899 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:47.657992 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:47.658512 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.659334 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.660334 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.660648 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.660751 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:47.661112 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:47.661179 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:47.661276 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:47.661408 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 04:49:47.661474 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:49:47.661782 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:47.656916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.4.217.60:41971)
0808 04:49:47.657638 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:47.657644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:47.657870 (+ 226us) server_negotiation.cc:407] Connection header received
0808 04:49:47.658056 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:47.658059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:47.658108 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:47.658160 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:47.658586 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.659178 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.660518 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.660777 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.660876 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:47.661275 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:47.661383 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:47.661452 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:47.661600 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 04:49:47.661653 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637}
I20260808 04:49:47.662686 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:47.656953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.4.217.62:35039)
0808 04:49:47.657463 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:47.657469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:47.657482 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:49:47.657583 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:47.657586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:47.657754 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:47.658177 (+ 423us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:47.658303 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.659138 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.660489 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:47.660855 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:47.660982 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:47.661626 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:47.661999 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:47.662414 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:47.662451 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 04:49:47.662511 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":73,"threads_started":1}
I20260808 04:49:47.665861 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 8947
I20260808 04:49:47.682389 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9073
I20260808 04:49:47.695717 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9197
2026-08-08T04:49:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3342 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:49:47 dist-test-slave-2kcd krb5kdc[9585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:49:47 dist-test-slave-2kcd krb5kdc[9585](info): set up 2 sockets
Aug 08 04:49:47 dist-test-slave-2kcd krb5kdc[9585](info): commencing operation
krb5kdc: starting...
W20260808 04:49:48.966898 4964 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/test-user.keytab.
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
Entry for principal kudu/127.4.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab.
I20260808 04:49:49.099447 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41751
--webserver_interface=127.4.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41751,127.4.217.61:38885,127.4.217.60:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:49:49.215003 9606 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:49:49.215286 9606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:49.215312 9606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:49.215409 9606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:49.219456 9606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:49.219523 9606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:49.219542 9606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:49.219559 9606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:49.219609 9606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:49.224879 9606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41751,127.4.217.61:38885,127.4.217.60:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.62.keytab
--principal=kudu/127.4.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:49.226027 9606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:49.227299 9606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:49.236173 9616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:49.236214 9622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:49.236428 9606 server_base.cc:1034] running on GCE node
W20260808 04:49:49.236582 9620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:49.237251 9606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:49.238267 9606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:49.239473 9606 hybrid_clock.cc:648] HybridClock initialized: now 1786164589239466 us; error 31 us; skew 500 ppm
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:49.242447 9606 init.cc:377] Logged in from keytab as kudu/127.4.217.62@KRBTEST.COM (short username kudu)
I20260808 04:49:49.243292 9606 webserver.cc:460] Webserver started at http://127.4.217.62:46071/ using document root <none> and password file <none>
I20260808 04:49:49.243817 9606 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:49.243875 9606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:49.244045 9606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:49.246150 9606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/data/instance:
uuid: "61851c865c8142da924bcfbfaf0270fd"
format_stamp: "Formatted at 2026-08-08 04:49:49 on dist-test-slave-2kcd"
server_key: "fc77fc1db0d905d8e77461a148cd87cb"
server_key_iv: "e474b19cfaca3f3bb10e57b2b2c9bdf8"
server_key_version: "encryptionkey@0"
I20260808 04:49:49.246768 9606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/wal/instance:
uuid: "61851c865c8142da924bcfbfaf0270fd"
format_stamp: "Formatted at 2026-08-08 04:49:49 on dist-test-slave-2kcd"
server_key: "fc77fc1db0d905d8e77461a148cd87cb"
server_key_iv: "e474b19cfaca3f3bb10e57b2b2c9bdf8"
server_key_version: "encryptionkey@0"
I20260808 04:49:49.250653 9606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:49:49.253172 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:49.254446 9606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:49.254578 9606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/wal
uuid: "61851c865c8142da924bcfbfaf0270fd"
format_stamp: "Formatted at 2026-08-08 04:49:49 on dist-test-slave-2kcd"
server_key: "fc77fc1db0d905d8e77461a148cd87cb"
server_key_iv: "e474b19cfaca3f3bb10e57b2b2c9bdf8"
server_key_version: "encryptionkey@0"
I20260808 04:49:49.254673 9606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:49.267704 9606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:49.271061 9606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:49.271224 9606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:49.278908 9606 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:41751
I20260808 04:49:49.279194 9708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.62:41751 every 8 connection(s)
I20260808 04:49:49.279902 9606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/data/info.pb
I20260808 04:49:49.282747 9709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:49.286403 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9606
I20260808 04:49:49.286590 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-0/wal/instance
I20260808 04:49:49.286392 9709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:49.286952 4964 external_mini_cluster.cc:1428] Setting key d65dd6379af32ff2cd5e4b8b62e7ade1
WARNING: no policy specified for kudu/127.4.217.61@KRBTEST.COM; defaulting to no policy
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.61@KRBTEST.COM" created.
I20260808 04:49:49.302327 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.288659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41751 (local address 127.0.0.1:50116)
0808 04:49:49.289176 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.289227 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:49.290112 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:49.290966 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:49.290970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:49.291304 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:49.291546 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.291558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.293434 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.293437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.294176 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.294187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.295177 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.295181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.295271 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.295879 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:49.295896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:49.297414 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:49.299236 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.299247 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.299257 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.299475 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.299815 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.299820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.299822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.299933 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.300574 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:49.300579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:49.300717 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 04:49:49.300959 (+ 242us) client_negotiation.cc:240] Negotiation successful
0808 04:49:49.301384 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":148,"thread_start_us":202,"threads_started":1}
I20260808 04:49:49.302327 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.289577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.4.217.62:41751)
0808 04:49:49.289919 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.289936 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.290002 (+ 66us) server_negotiation.cc:407] Connection header received
0808 04:49:49.290439 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.290448 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.290707 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.290922 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.291712 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.293284 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.294319 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.294975 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.295179 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.297563 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.297569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.297579 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.297603 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.299117 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.299596 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.299604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.299606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.299710 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.300040 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.300045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.300046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.300442 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.300602 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.301033 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.301332 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260808 04:49:49.304647 9709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:49.305418 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.305165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38885 (local address 127.0.0.1:44780)
0808 04:49:49.305282 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.305360 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:49:49.305629 9632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
I20260808 04:49:49.306526 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.306307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38885 (local address 127.0.0.1:44790)
0808 04:49:49.306415 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.306460 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:49:49.306653 9709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.61:38885: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab.
Entry for principal kudu/127.4.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab.
I20260808 04:49:49.316673 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:38885
--webserver_interface=127.4.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab
--principal=kudu/127.4.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41751,127.4.217.61:38885,127.4.217.60:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:49:49.355885 9709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } attempt: 1
I20260808 04:49:49.356993 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.356637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38885 (local address 127.0.0.1:44796)
0808 04:49:49.356819 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.356912 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 04:49:49.357738 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.357518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38885 (local address 127.0.0.1:44800)
0808 04:49:49.357629 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.357683 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:49:49.357854 9709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.61:38885: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
I20260808 04:49:49.426035 9709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } attempt: 2
I20260808 04:49:49.427011 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.426673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38885 (local address 127.0.0.1:44808)
0808 04:49:49.426848 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.426929 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 04:49:49.427806 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.427566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38885 (local address 127.0.0.1:44810)
0808 04:49:49.427690 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.427746 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 04:49:49.427937 9709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.61:38885: Network error: Client connection negotiation failed: client connection to 127.4.217.61:38885: connect: Connection refused (error 111)
W20260808 04:49:49.433295 9726 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:49:49.433627 9726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:49.433701 9726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:49.433805 9726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:49.437885 9726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:49.437973 9726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:49.438024 9726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:49.438088 9726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:49.438130 9726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:49.443593 9726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41751,127.4.217.61:38885,127.4.217.60:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.61.keytab
--principal=kudu/127.4.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:49.444705 9726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:49.445770 9726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:49:49.454558 9736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:49.454598 9734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:49.454612 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:49.454962 9726 server_base.cc:1034] running on GCE node
I20260808 04:49:49.455449 9726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:49.456486 9726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:49.457687 9726 hybrid_clock.cc:648] HybridClock initialized: now 1786164589457629 us; error 47 us; skew 500 ppm
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:49.460669 9726 init.cc:377] Logged in from keytab as kudu/127.4.217.61@KRBTEST.COM (short username kudu)
I20260808 04:49:49.461613 9726 webserver.cc:460] Webserver started at http://127.4.217.61:43541/ using document root <none> and password file <none>
I20260808 04:49:49.462208 9726 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:49.462328 9726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:49.462579 9726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:49.464877 9726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/data/instance:
uuid: "f24256b2987b4bad83661d99404545be"
format_stamp: "Formatted at 2026-08-08 04:49:49 on dist-test-slave-2kcd"
server_key: "63ead4dfdb6ce082bfc2be49db611200"
server_key_iv: "9379f550f8aa26180fee43ee8454f5be"
server_key_version: "encryptionkey@0"
I20260808 04:49:49.465481 9726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/wal/instance:
uuid: "f24256b2987b4bad83661d99404545be"
format_stamp: "Formatted at 2026-08-08 04:49:49 on dist-test-slave-2kcd"
server_key: "63ead4dfdb6ce082bfc2be49db611200"
server_key_iv: "9379f550f8aa26180fee43ee8454f5be"
server_key_version: "encryptionkey@0"
I20260808 04:49:49.469303 9726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:49:49.471693 9750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:49.473047 9726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:49:49.473207 9726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/wal
uuid: "f24256b2987b4bad83661d99404545be"
format_stamp: "Formatted at 2026-08-08 04:49:49 on dist-test-slave-2kcd"
server_key: "63ead4dfdb6ce082bfc2be49db611200"
server_key_iv: "9379f550f8aa26180fee43ee8454f5be"
server_key_version: "encryptionkey@0"
I20260808 04:49:49.473325 9726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:49.483374 9726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:49.486373 9726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:49.486588 9726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:49.493778 9726 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:38885
I20260808 04:49:49.494081 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.61:38885 every 8 connection(s)
I20260808 04:49:49.494803 9726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/data/info.pb
I20260808 04:49:49.497736 9842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:49.501542 9842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:49.503101 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9726
I20260808 04:49:49.503273 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-1/wal/instance
I20260808 04:49:49.503665 4964 external_mini_cluster.cc:1428] Setting key 49c0fef5f146caa895e89463f14b382a
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.217.60@KRBTEST.COM; defaulting to no policy
I20260808 04:49:49.516886 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.503483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.4.217.62:41751)
0808 04:49:49.503661 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.503666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.504606 (+ 940us) server_negotiation.cc:407] Connection header received
0808 04:49:49.505580 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.505587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.505668 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.505794 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.507514 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.508080 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.509349 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.509822 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.509986 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.512654 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.512661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.512664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.512700 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.514502 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.515148 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.515151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.515152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.515232 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.515561 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.515563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.515564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.515747 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.515826 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.516542 (+ 716us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.516663 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:49:49.518117 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.503909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41751 (local address 127.0.0.1:50124)
0808 04:49:49.504409 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.504464 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:49.505355 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:49.506051 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:49.506062 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:49.506600 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:49.507323 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.507344 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.508205 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.508207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.509209 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.509219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.510068 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.510075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.510284 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.510955 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:49.510972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:49.512463 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:49.514632 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.514638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.514651 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.515029 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.515325 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.515327 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.515328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.515473 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.515872 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:49.515903 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:49.516265 (+ 362us) client_negotiation.cc:769] Sending connection context
0808 04:49:49.516444 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 04:49:49.516730 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":157,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.217.60@KRBTEST.COM" created.
I20260808 04:49:49.519111 9709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } attempt: 3
I20260808 04:49:49.519897 9842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
I20260808 04:49:49.531896 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.520423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38885 (local address 127.0.0.1:44826)
0808 04:49:49.520808 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.520822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:49.520889 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:49.521456 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:49.521460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:49.521625 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:49.521726 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.521732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.522765 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.522769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.523670 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.523681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.524537 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.524542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.524624 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.525386 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:49.525401 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:49.527062 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:49.529776 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.529779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.529780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.530251 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.530584 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.530588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.530590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.530729 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.531336 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:49.531342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:49.531459 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 04:49:49.531584 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 04:49:49.531746 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300}
I20260808 04:49:49.532079 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.520547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.4.217.61:38885)
0808 04:49:49.521157 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.521167 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.521186 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:49:49.521226 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.521230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.521316 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.521422 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.521858 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.522649 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.523812 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.524306 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.524448 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.527234 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.527243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.527255 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.527275 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.529627 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.530374 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.530381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.530383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.530469 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.530860 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.530863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.530864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.531236 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.531379 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.531699 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.531853 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"spinlock_wait_cycles":4352,"thread_start_us":119,"threads_started":1}
I20260808 04:49:49.532235 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.519574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38885 (local address 127.0.0.1:44820)
0808 04:49:49.519755 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.519770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:49.519851 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:49.520807 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:49.520810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:49.521008 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:49.521117 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.521122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.522426 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.522432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.523762 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.523774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.524471 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.524476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.524587 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.525386 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:49.525401 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:49.527770 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:49.530584 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.530587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.530590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.530871 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.531147 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.531149 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.531151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.531294 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.531719 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:49.531722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:49.531826 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 04:49:49.531912 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 04:49:49.532046 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 04:49:49.533061 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.519814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.4.217.61:38885)
0808 04:49:49.520297 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.520311 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.520336 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:49:49.520386 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.520392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.520614 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.521244 (+ 630us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.521363 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.522301 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.523873 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.524233 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.524363 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.527949 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.527954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.527959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.527988 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.530466 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.530965 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.530968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.530969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.531048 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.531380 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.531384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.531387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.531596 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.532693 (+ 1097us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.532755 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.532914 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":361,"spinlock_wait_cycles":2816,"thread_start_us":104,"threads_started":1}
I20260808 04:49:49.534426 9709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:49.534464 9842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:49.535246 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.534979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57942)
0808 04:49:49.535121 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.535178 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:49:49.535274 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.535002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57936)
0808 04:49:49.535144 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.535223 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 04:49:49.535714 9752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
I20260808 04:49:49.536528 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.536190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57946)
0808 04:49:49.536391 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.536448 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260808 04:49:49.536775 9709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:34457: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
I20260808 04:49:49.537269 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.537060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57956)
0808 04:49:49.537169 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.537217 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:49:49.537484 9842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:34457: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab.
Entry for principal kudu/127.4.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab.
I20260808 04:49:49.540542 4964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskywCe24/build/debug/bin/kudu
/tmp/dist-test-taskywCe24/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:34457
--webserver_interface=127.4.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab
--principal=kudu/127.4.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.217.20:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41751,127.4.217.61:38885,127.4.217.60:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:49:49.568028 9709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } attempt: 1
I20260808 04:49:49.569062 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.568679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57968)
0808 04:49:49.568900 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.568973 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 04:49:49.569960 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.569682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57970)
0808 04:49:49.569799 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.569892 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:49:49.570150 9709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:34457: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
I20260808 04:49:49.586755 9842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } attempt: 1
I20260808 04:49:49.587720 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.587336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57972)
0808 04:49:49.587522 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.587614 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 04:49:49.588610 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.588376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57976)
0808 04:49:49.588491 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.588536 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:49:49.588841 9842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:34457: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
I20260808 04:49:49.645335 9709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } attempt: 2
I20260808 04:49:49.646301 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.645955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57984)
0808 04:49:49.646147 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.646214 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 04:49:49.647133 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.646888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:57994)
0808 04:49:49.647016 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.647072 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:49:49.647272 9709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:34457: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
I20260808 04:49:49.657017 9842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } attempt: 2
I20260808 04:49:49.657814 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.657511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:58010)
0808 04:49:49.657648 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.657731 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:49:49.658622 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.658335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:58020)
0808 04:49:49.658476 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.658560 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 04:49:49.658751 9842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.217.60:34457: Network error: Client connection negotiation failed: client connection to 127.4.217.60:34457: connect: Connection refused (error 111)
W20260808 04:49:49.659600 9851 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:49:49.659902 9851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:49:49.659966 9851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:49:49.660038 9851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:49:49.664762 9851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:49:49.664870 9851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:49:49.664909 9851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:49:49.664947 9851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:49:49.664983 9851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:49:49.671103 9851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41751,127.4.217.61:38885,127.4.217.60:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/krb5kdc/kudu_127.4.217.60.keytab
--principal=kudu/127.4.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:34457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:49:49.672120 9851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:49:49.673173 9851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:49:49.682643 9851 server_base.cc:1034] running on GCE node
W20260808 04:49:49.682571 9862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:49.682691 9864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:49:49.682579 9861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:49:49.683491 9851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:49:49.684614 9851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:49:49.685824 9851 hybrid_clock.cc:648] HybridClock initialized: now 1786164589685783 us; error 58 us; skew 500 ppm
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:49:49.688844 9851 init.cc:377] Logged in from keytab as kudu/127.4.217.60@KRBTEST.COM (short username kudu)
I20260808 04:49:49.689770 9851 webserver.cc:460] Webserver started at http://127.4.217.60:46613/ using document root <none> and password file <none>
I20260808 04:49:49.690358 9851 fs_manager.cc:359] Metadata directory not provided
I20260808 04:49:49.690457 9851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:49:49.690702 9851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:49:49.692756 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/data/instance:
uuid: "57cdf848e8064dc1bcc17c787959daaf"
format_stamp: "Formatted at 2026-08-08 04:49:49 on dist-test-slave-2kcd"
server_key: "a340ec13ced49f74bcb50772fa9d58ae"
server_key_iv: "6e998625041b1a1fa191aacf51f3de84"
server_key_version: "encryptionkey@0"
I20260808 04:49:49.693405 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/wal/instance:
uuid: "57cdf848e8064dc1bcc17c787959daaf"
format_stamp: "Formatted at 2026-08-08 04:49:49 on dist-test-slave-2kcd"
server_key: "a340ec13ced49f74bcb50772fa9d58ae"
server_key_iv: "6e998625041b1a1fa191aacf51f3de84"
server_key_version: "encryptionkey@0"
I20260808 04:49:49.697108 9851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:49:49.699543 9874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:49:49.700716 9851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:49:49.700901 9851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/data,/tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/wal
uuid: "57cdf848e8064dc1bcc17c787959daaf"
format_stamp: "Formatted at 2026-08-08 04:49:49 on dist-test-slave-2kcd"
server_key: "a340ec13ced49f74bcb50772fa9d58ae"
server_key_iv: "6e998625041b1a1fa191aacf51f3de84"
server_key_version: "encryptionkey@0"
I20260808 04:49:49.701021 9851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:49:49.725839 9851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:49:49.729322 9851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:49:49.729539 9851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:49:49.736879 9851 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.60:34457
I20260808 04:49:49.737243 9971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.217.60:34457 every 8 connection(s)
I20260808 04:49:49.738029 9851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/data/info.pb
I20260808 04:49:49.738364 4964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskywCe24/build/debug/bin/kudu as pid 9851
I20260808 04:49:49.738521 4964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskywCe24/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164296366965-4964-0/minicluster-data/master-2/wal/instance
I20260808 04:49:49.738832 4964 external_mini_cluster.cc:1428] Setting key 896ac639e4feb55e969f2d58d0b77284
I20260808 04:49:49.741493 9973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:49:49.745325 9973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:49.746454 9709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } attempt: 3
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
I20260808 04:49:49.749917 9842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } attempt: 3
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.62@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.61@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
I20260808 04:49:49.756254 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.740265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.4.217.60:34457)
0808 04:49:49.740741 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.740756 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.740840 (+ 84us) server_negotiation.cc:407] Connection header received
0808 04:49:49.741521 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.741541 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.741930 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.742262 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.743096 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.744116 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.745043 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.745505 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.745674 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.749430 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.749439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.749451 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.749481 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.752111 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.752923 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.752935 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.752943 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.753095 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.753460 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.753464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.753466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.754011 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.754221 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.754766 (+ 545us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.755163 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":144,"threads_started":1}
I20260808 04:49:49.759433 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.746925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:58026)
0808 04:49:49.747145 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.747160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:49.747250 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:49.748002 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:49.748005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:49.748198 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:49.748348 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.748358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.749338 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.749343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.750185 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.750196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.750852 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.750856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.750955 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.751719 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:49.751734 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:49.753388 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:49.757375 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.757382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.757385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.757656 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.758170 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.758173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.758175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.758283 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.758795 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:49.758800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:49.758921 (+ 121us) client_negotiation.cc:769] Sending connection context
0808 04:49:49.759100 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 04:49:49.759242 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260808 04:49:49.760016 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.747462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.4.217.60:34457)
0808 04:49:49.747741 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.747750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.747763 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:49:49.747799 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.747804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.747852 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.748468 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.748609 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.749224 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.750303 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.750668 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.750826 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.753540 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.753543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.753545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.753574 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.757239 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.757746 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.757751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.757753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.757841 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.758415 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.758419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.758420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.758642 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.758746 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.759635 (+ 889us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.759765 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":88,"threads_started":1}
I20260808 04:49:49.761462 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.750437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:58038)
0808 04:49:49.750896 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.750911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:49.750990 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:49.751276 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:49.751278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:49.751462 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:49.751568 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.751578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.752854 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.752858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.753667 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.753677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.754403 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.754410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.754514 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.754945 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:49.754961 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:49.756615 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:49.758865 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.758871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.758875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.759144 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.760080 (+ 936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.760085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.760090 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.760192 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.760931 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:49.760934 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:49.761036 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 04:49:49.761151 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 04:49:49.761301 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260808 04:49:49.761950 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.750448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.4.217.60:34457)
0808 04:49:49.751050 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.751057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.751068 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:49:49.751098 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.751101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.751136 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.751677 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.751777 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.752672 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.753822 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.754226 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.754766 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.756763 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.756767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.756769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.756823 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.758695 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.759307 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.759312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.759314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.759409 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.760559 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.760563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.760566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.760826 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.760925 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.761670 (+ 745us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.761783 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":530,"thread_start_us":89,"threads_started":1}
I20260808 04:49:49.763242 9709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd: Bootstrap starting.
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.62@KRBTEST.COM
I20260808 04:49:49.766012 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.747980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.4.217.62:41751)
0808 04:49:49.748087 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.748092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.748107 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:49:49.748278 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.748281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.748319 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.748406 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.749451 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.750162 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.751665 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.752018 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.752126 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.756864 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.756868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.756870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.756897 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.758834 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.759440 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.759447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.759452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.759535 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.759962 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.759965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.759967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.760208 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.760285 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.765647 (+ 5362us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.765786 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:49:49.766109 9709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:49.766690 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.747077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41751 (local address 127.0.0.1:50138)
0808 04:49:49.747995 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.748058 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:49.748168 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:49.748509 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:49.748516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:49.749081 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:49.749262 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.749275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.750303 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.750306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.751517 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.751527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.752246 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.752250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.752413 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.753018 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:49.753031 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:49.756680 (+ 3649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:49.758945 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.758955 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.758960 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.759318 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.759765 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.759768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.759769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.759852 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.760385 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:49.760391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:49.760510 (+ 119us) client_negotiation.cc:769] Sending connection context
0808 04:49:49.766211 (+ 5701us) client_negotiation.cc:240] Negotiation successful
0808 04:49:49.766468 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":303,"thread_start_us":151,"threads_started":1}
I20260808 04:49:49.767112 9709 log.cc:824] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:49.767925 9973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:49.768955 9842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be: Bootstrap starting.
I20260808 04:49:49.769508 9709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd: No bootstrap required, opened a new log
I20260808 04:49:49.769937 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.759656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.4.217.62:41751)
0808 04:49:49.759937 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.759951 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.759966 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:49:49.760009 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.760014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.760065 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.760145 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.761078 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.761828 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.762808 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.763053 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.763162 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.765802 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.765805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.765808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.765831 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.767862 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.768421 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.768425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.768426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.768520 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.768892 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.768896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.768897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.769187 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.769379 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.769553 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.769737 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260808 04:49:49.771682 9842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:49.772612 9842 log.cc:824] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:49.774575 9709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } }
I20260808 04:49:49.774829 9709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61851c865c8142da924bcfbfaf0270fd, State: Initialized, Role: FOLLOWER
I20260808 04:49:49.775199 9709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } }
I20260808 04:49:49.775714 9988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.775943 9988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:49.776736 9709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.217.61@KRBTEST.COM
I20260808 04:49:49.781713 9842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be: No bootstrap required, opened a new log
W20260808 04:49:49.782930 10004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:49:49.783064 10004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:49:49.783156 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.768419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:38885 (local address 127.0.0.1:44842)
0808 04:49:49.768763 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.768807 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:49.768880 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:49.773332 (+ 4452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:49.773336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:49.773527 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:49.773628 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.773634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.774586 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.774593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.775480 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.775495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.776370 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.776381 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.776476 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.777201 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:49.777216 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:49.778924 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:49.781241 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.781244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.781246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.781515 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.781857 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.781860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.781862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.781959 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.782502 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:49.782505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:49.782617 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 04:49:49.782726 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:49:49.782876 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":107,"threads_started":1}
I20260808 04:49:49.783742 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.772979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.4.217.61:38885)
0808 04:49:49.773131 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.773134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.773146 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:49:49.773178 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.773180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.773214 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.773282 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.773746 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.774442 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.775658 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.776258 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.776330 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.779115 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.779122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.779126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.779157 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.781111 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.781643 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.781646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.781648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.781743 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.782075 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.782081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.782082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.782351 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.782442 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.783402 (+ 960us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.783563 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":309,"server-negotiator.queue_time_us":74}
I20260808 04:49:49.785063 9973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:49:49.787079 9842 raft_consensus.cc:348] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } }
I20260808 04:49:49.787341 9842 raft_consensus.cc:729] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24256b2987b4bad83661d99404545be, State: Initialized, Role: FOLLOWER
I20260808 04:49:49.788245 9842 consensus_queue.cc:260] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } }
I20260808 04:49:49.788411 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.773015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.4.217.61:38885)
0808 04:49:49.773138 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.773144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.773161 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:49:49.773191 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.773193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.773226 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.773322 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.774084 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.774699 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.775608 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.775970 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.776056 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.784458 (+ 8402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.784462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.784464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.784493 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.786284 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.786866 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.786871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.786876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.786974 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.787319 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.787322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.787324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.787634 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.787843 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.788075 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.788205 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":53}
I20260808 04:49:49.789820 9842 sys_catalog.cc:564] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:49.783552 9606 master_runner.cc:186] Error getting master registration for 127.4.217.61:38885: OK, instance_id { permanent_uuid: "f24256b2987b4bad83661d99404545be" instance_seqno: 1786164589491706 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:49:49.790597 9606 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:49:49.791991 4964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:49:49.793380 9676 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "61851c865c8142da924bcfbfaf0270fd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146
I20260808 04:49:49.793663 9676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 04:49:49.793788 9676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [term 0 FOLLOWER]: Advancing to term 1
Aug 08 04:49:49 dist-test-slave-2kcd krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.217.60@KRBTEST.COM for kudu/127.4.217.60@KRBTEST.COM
I20260808 04:49:49.795961 9676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } }
I20260808 04:49:49.796298 10009 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.796490 10009 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:49.799724 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.785888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.60:34457 (local address 127.0.0.1:58044)
0808 04:49:49.786292 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0808 04:49:49.786306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:49:49.786386 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:49:49.786710 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:49:49.786713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:49:49.786900 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:49:49.787022 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.787030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.790716 (+ 3686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.790720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.791489 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:49:49.791513 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.792191 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.792194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:49:49.792261 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.792758 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:49:49.792772 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:49:49.794424 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:49:49.798132 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.798135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.798137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.798384 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.798671 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:49:49.798673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:49:49.798675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:49:49.798756 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:49:49.799196 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:49:49.799198 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:49:49.799291 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 04:49:49.799407 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:49:49.799569 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359}
W20260808 04:49:49.798137 10032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 04:49:49.800345 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.786114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.4.217.60:34457)
0808 04:49:49.786231 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.786236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.786460 (+ 224us) server_negotiation.cc:407] Connection header received
0808 04:49:49.786489 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.786495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.786544 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.786607 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.787135 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.787866 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.791629 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.791988 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.792089 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.794568 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.794572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.794574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.794596 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.798014 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.798497 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.798500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.798502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.798579 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.798844 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.798847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.798848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.799071 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.799163 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.800057 (+ 894us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.800196 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 04:49:49.800866 10032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:49:49.803567 9973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf: Bootstrap starting.
I20260808 04:49:49.800321 9726 master_runner.cc:186] Error getting master registration for 127.4.217.60:34457: OK, instance_id { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" instance_seqno: 1786164589734720 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:49:49.803815 9726 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:49:49.805781 9676 leader_election.cc:288] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [CANDIDATE]: Term 1 election: Requested vote from peers f24256b2987b4bad83661d99404545be (127.4.217.61:38885), 57cdf848e8064dc1bcc17c787959daaf (127.4.217.60:34457)
I20260808 04:49:49.806874 9973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf: Neither blocks nor log segments found. Creating new log.
I20260808 04:49:49.806923 9795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61851c865c8142da924bcfbfaf0270fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f24256b2987b4bad83661d99404545be"
I20260808 04:49:49.807201 9795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:49.807835 9973 log.cc:824] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf: Log is configured to *not* fsync() on all Append() calls
I20260808 04:49:49.808606 9795 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61851c865c8142da924bcfbfaf0270fd in term 1.
I20260808 04:49:49.809113 9632 leader_election.cc:302] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61851c865c8142da924bcfbfaf0270fd, f24256b2987b4bad83661d99404545be; no voters:
I20260808 04:49:49.809465 9988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:49:49.809686 9988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [term 1 LEADER]: Becoming Leader. State: Replica: 61851c865c8142da924bcfbfaf0270fd, State: Running, Role: LEADER
I20260808 04:49:49.810081 9988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } }
I20260808 04:49:49.811982 9988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61851c865c8142da924bcfbfaf0270fd. Latest consensus state: current_term: 1 leader_uuid: "61851c865c8142da924bcfbfaf0270fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.812116 9988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:49.819168 9973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf: No bootstrap required, opened a new log
I20260808 04:49:49.824666 9973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } }
I20260808 04:49:49.825212 9973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cdf848e8064dc1bcc17c787959daaf, State: Initialized, Role: FOLLOWER
I20260808 04:49:49.825646 9973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } }
I20260808 04:49:49.827379 9973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:49:49.827446 9922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61851c865c8142da924bcfbfaf0270fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "57cdf848e8064dc1bcc17c787959daaf"
I20260808 04:49:49.827703 9922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:49:49.829200 10047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:49:49.829324 9922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61851c865c8142da924bcfbfaf0270fd in term 1.
I20260808 04:49:49.830451 10047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:49:49.839402 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.817725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.4.217.61:38885)
0808 04:49:49.817868 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.817881 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.818765 (+ 884us) server_negotiation.cc:407] Connection header received
0808 04:49:49.819196 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.819199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.819252 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.819302 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.822192 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.822977 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.832599 (+ 9622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.832965 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.833056 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.835024 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.835027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.835030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.835058 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.836854 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.837391 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.837395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.837397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.837481 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.837793 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.837796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.837798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.838038 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.838398 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.839079 (+ 681us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.839224 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:49:49.842033 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.842295 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:49.843497 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.817277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.4.217.62:41751)
0808 04:49:49.817494 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.817501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.818477 (+ 976us) server_negotiation.cc:407] Connection header received
0808 04:49:49.818614 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.818617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.818731 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.818859 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.824317 (+ 5458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.825544 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.837275 (+ 11731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.837673 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.837785 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.839547 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.839552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.839558 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.839593 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.841310 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.841811 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.841815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.841817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.841912 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.842240 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.842244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.842245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.842506 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.842620 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.843188 (+ 568us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.843336 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:49:49.844645 9922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [term 1 FOLLOWER]: Refusing update from remote peer 61851c865c8142da924bcfbfaf0270fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:49:49.845306 10039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:49.846810 9795 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [term 1 FOLLOWER]: Refusing update from remote peer 61851c865c8142da924bcfbfaf0270fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 04:49:49.847292 10083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:49:49.847360 10083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:49:49.847355 10039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:49:49.849990 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61851c865c8142da924bcfbfaf0270fd. Latest consensus state: current_term: 1 leader_uuid: "61851c865c8142da924bcfbfaf0270fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.850121 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:49.850641 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:49.818111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.4.217.60:34457)
0808 04:49:49.818225 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.818230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.818972 (+ 742us) server_negotiation.cc:407] Connection header received
0808 04:49:49.819157 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.819161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.819215 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.819305 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.826019 (+ 6714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.826744 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.833355 (+ 6611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:49.833703 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:49.833822 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:49.845104 (+ 11282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:49:49.845108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:49:49.845110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:49:49.845140 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:49:49.848287 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.848972 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.848975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.848977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.849070 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:49:49.849411 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:49:49.849416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:49:49.849418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:49:49.849668 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:49:49.849841 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:49.850333 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 04:49:49.850469 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:49:49.851831 10009 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61851c865c8142da924bcfbfaf0270fd. Latest consensus state: current_term: 1 leader_uuid: "61851c865c8142da924bcfbfaf0270fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.851954 10009 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:49.854753 10039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61851c865c8142da924bcfbfaf0270fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.854885 10039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:49.856112 10009 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61851c865c8142da924bcfbfaf0270fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.856235 10009 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:49.856648 10084 mvcc.cc:205] Tried to move back new op lower bound from 7316130159995453440 to 7316130159866187776. Current Snapshot: MvccSnapshot[applied={T|T < 7316130159995453440}]
I20260808 04:49:49.857707 10089 mvcc.cc:205] Tried to move back new op lower bound from 7316130159995453440 to 7316130159866187776. Current Snapshot: MvccSnapshot[applied={T|T < 7316130159995453440}]
I20260808 04:49:49.858876 10047 catalog_manager.cc:1347] Generated new cluster ID: a48793dce0424fd4a03721e5533555fb
I20260808 04:49:49.858947 10047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:49:49.865231 9988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61851c865c8142da924bcfbfaf0270fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.865357 9988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd [sys.catalog]: This master's current role is: LEADER
I20260808 04:49:49.866634 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61851c865c8142da924bcfbfaf0270fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61851c865c8142da924bcfbfaf0270fd" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41751 } } peers { permanent_uuid: "f24256b2987b4bad83661d99404545be" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 38885 } } peers { permanent_uuid: "57cdf848e8064dc1bcc17c787959daaf" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 34457 } } }
I20260808 04:49:49.866753 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:49:49.890177 10047 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:49:49.891866 10047 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:49:49.902268 10047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61851c865c8142da924bcfbfaf0270fd: Generated new TSK 0
I20260808 04:49:49.903228 10047 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:49:49.936550 9980 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:49.933286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.4.217.60:34457)
0808 04:49:49.933416 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.933421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.935368 (+ 1947us) server_negotiation.cc:407] Connection header received
0808 04:49:49.935583 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.935587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.935642 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.935696 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.936396 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58070: BlockingRecv error: recv got EOF from 127.0.0.1:58070 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 04:49:49.938067 9982 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:49.934133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.4.217.62:41751)
0808 04:49:49.934253 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.934257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.937120 (+ 2863us) server_negotiation.cc:407] Connection header received
0808 04:49:49.937306 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.937309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.937361 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.937412 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.937928 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50170: BlockingRecv error: recv got EOF from 127.0.0.1:50170 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 04:49:49.939489 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:49.934770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.4.217.61:38885)
0808 04:49:49.934887 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.934891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.938564 (+ 3673us) server_negotiation.cc:407] Connection header received
0808 04:49:49.938739 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.938742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.938796 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.938847 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.939363 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44874: BlockingRecv error: recv got EOF from 127.0.0.1:44874 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260808 04:49:49.944092 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:49.942588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.4.217.61:38885)
0808 04:49:49.942695 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.942699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.943137 (+ 438us) server_negotiation.cc:407] Connection header received
0808 04:49:49.943352 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.943355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.943431 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.943482 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.943975 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44886: BlockingRecv error: recv got EOF from 127.0.0.1:44886 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 04:49:49.949295 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:49.947723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.4.217.61:38885)
0808 04:49:49.947823 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.947827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.948297 (+ 470us) server_negotiation.cc:407] Connection header received
0808 04:49:49.948472 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.948476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.948525 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.948595 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.949199 (+ 604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44900: BlockingRecv error: recv got EOF from 127.0.0.1:44900 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 04:49:49.954949 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:49.953513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.4.217.61:38885)
0808 04:49:49.953608 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.953612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.954088 (+ 476us) server_negotiation.cc:407] Connection header received
0808 04:49:49.954274 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.954277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.954325 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.954374 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.954853 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44910: BlockingRecv error: recv got EOF from 127.0.0.1:44910 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 04:49:49.963145 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:49.961509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.4.217.61:38885)
0808 04:49:49.961657 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.961662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.962183 (+ 521us) server_negotiation.cc:407] Connection header received
0808 04:49:49.962378 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.962381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.962435 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.962489 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.963040 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44926: BlockingRecv error: recv got EOF from 127.0.0.1:44926 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260808 04:49:49.975281 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:49.973644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.4.217.61:38885)
0808 04:49:49.973754 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.973759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.974278 (+ 519us) server_negotiation.cc:407] Connection header received
0808 04:49:49.974477 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.974480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.974536 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.974591 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.975161 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44928: BlockingRecv error: recv got EOF from 127.0.0.1:44928 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 04:49:49.995937 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:49.994112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.4.217.61:38885)
0808 04:49:49.994245 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:49.994250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:49.994842 (+ 592us) server_negotiation.cc:407] Connection header received
0808 04:49:49.995079 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:49.995083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:49.995145 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:49.995200 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:49.995807 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44936: BlockingRecv error: recv got EOF from 127.0.0.1:44936 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 04:49:50.031960 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:50.030773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.4.217.61:38885)
0808 04:49:50.030927 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:50.030932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:50.031082 (+ 150us) server_negotiation.cc:407] Connection header received
0808 04:49:50.031178 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:50.031182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:50.031248 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:50.031361 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:50.031823 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44948: BlockingRecv error: recv got EOF from 127.0.0.1:44948 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 04:49:50.099233 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:50.098129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.4.217.61:38885)
0808 04:49:50.098277 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:50.098281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:50.098437 (+ 156us) server_negotiation.cc:407] Connection header received
0808 04:49:50.098557 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:50.098560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:50.098608 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:50.098706 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:50.099135 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44950: BlockingRecv error: recv got EOF from 127.0.0.1:44950 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 04:49:50.230902 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:50.229625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.4.217.61:38885)
0808 04:49:50.229797 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:50.229802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:50.229945 (+ 143us) server_negotiation.cc:407] Connection header received
0808 04:49:50.230151 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:50.230154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:50.230212 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:50.230304 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:50.230777 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44954: BlockingRecv error: recv got EOF from 127.0.0.1:44954 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 04:49:50.490365 9849 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:49:50.489055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.4.217.61:38885)
0808 04:49:50.489204 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:50.489210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:50.489395 (+ 185us) server_negotiation.cc:407] Connection header received
0808 04:49:50.489578 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:50.489582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:50.489640 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:50.489732 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:49:50.490259 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46636: BlockingRecv error: recv got EOF from 127.0.0.1:46636 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:49:50.802428 10032 catalog_manager.cc:1259] Loaded cluster ID: a48793dce0424fd4a03721e5533555fb
I20260808 04:49:50.802522 10032 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be: loading cluster ID for follower catalog manager: success
I20260808 04:49:50.804337 10032 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be: acquiring CA information for follower catalog manager: success
I20260808 04:49:50.805631 10032 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f24256b2987b4bad83661d99404545be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:49:50.848666 10083 catalog_manager.cc:1259] Loaded cluster ID: a48793dce0424fd4a03721e5533555fb
I20260808 04:49:50.848742 10083 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf: loading cluster ID for follower catalog manager: success
I20260808 04:49:50.850411 10083 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf: acquiring CA information for follower catalog manager: success
I20260808 04:49:50.851543 10083 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 57cdf848e8064dc1bcc17c787959daaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:49:51.008865 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:51.004715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.4.217.61:38885)
0808 04:49:51.005068 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:51.005074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:51.005091 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:49:51.005248 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:51.005252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:51.005403 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:51.005540 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:51.005806 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.006362 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.007235 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.007467 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.007579 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:51.007903 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:51.008356 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:51.008494 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:51.008576 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 04:49:51.008635 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":139,"threads_started":1}
I20260808 04:49:51.015168 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:51.011799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.4.217.60:34457)
0808 04:49:51.012005 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:51.012009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:51.012021 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:49:51.012104 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:51.012107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:51.012214 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:51.012312 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:51.012503 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.013067 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.013836 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.014048 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.014121 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:51.014365 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:51.014766 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:51.014854 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:51.014916 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 04:49:51.014962 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":56,"threads_started":1}
I20260808 04:49:51.022614 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:51.018715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.4.217.61:38885)
0808 04:49:51.018844 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:51.018850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:51.019083 (+ 233us) server_negotiation.cc:407] Connection header received
0808 04:49:51.019324 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:51.019327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:51.019383 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:51.019458 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:51.019717 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.020353 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.021349 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.021680 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.021778 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:51.022118 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:51.022198 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:51.022265 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:51.022385 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 04:49:51.022439 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 04:49:51.022758 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:51.018288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47878 (local address 127.4.217.60:34457)
0808 04:49:51.018411 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:51.018417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:51.018702 (+ 285us) server_negotiation.cc:407] Connection header received
0808 04:49:51.018882 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:51.018886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:51.018937 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:51.019036 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:51.019351 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.020076 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.021315 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.021663 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.021778 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:51.022052 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:51.022133 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:51.022206 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:51.022562 (+ 356us) server_negotiation.cc:299] Negotiation successful
0808 04:49:51.022619 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:49:51.023240 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:49:51.018191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.4.217.62:41751)
0808 04:49:51.018584 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0808 04:49:51.018589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:49:51.018605 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:49:51.018705 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:49:51.018708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:49:51.018871 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:49:51.018988 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:49:51.019239 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.019925 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.021239 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:49:51.021518 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:49:51.021626 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:49:51.021958 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:49:51.022307 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:49:51.022384 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:49:51.022717 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 04:49:51.022781 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":75,"threads_started":1}
I20260808 04:49:51.026495 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9606
I20260808 04:49:51.041886 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9726
I20260808 04:49:51.060614 4964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskywCe24/build/debug/bin/kudu with pid 9851
2026-08-08T04:49:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3359 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6702 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (294687 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST